cloudfoundry / cf-k8s-logging

Apache License 2.0
5 stars 11 forks source link

Task logs don't come out if there's only one task #35

Open MasslessParticle opened 4 years ago

MasslessParticle commented 4 years ago

This might be because the task is super short lived. How long does it take for fluent tail to recognize new containers?

cf-gitbot commented 4 years ago

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

ericpromislow commented 4 years ago

In honor of today's smoky skies:

In terminal 1:

for x in {10..20} ; do echo $x; cf run-task node-app1 -c smoggie$x; done
10
Creating task for app node-app1 in org org / space space as admin...
Task has been submitted successfully for execution.
OK
...

In terminal 2:

cf logs node-app1
Retrieving logs for app node-app1 in org org / space space as admin...
   2020-09-08T10:03:18.50-0700 [TASK/8c477cec-8b4e-4c07-b5d7-686a57e918ea] OUT /bin/sh: 1: smoggie10: not found
   2020-09-08T10:03:21.68-0700 [TASK/812641c5-c292-43b1-87dc-93deaf8c2fa9] OUT /bin/sh: 1: smoggie12: not found
   2020-09-08T10:03:21.77-0700 [TASK/920ea315-d44f-44b5-9e60-869fffad8aa7] OUT /bin/sh: 1: smoggie11: not found
   2020-09-08T10:03:24.82-0700 [TASK/5bfbe4a4-c39d-4ead-a5c9-1078cc615564] OUT /bin/sh: 1: smoggie14: not found
   2020-09-08T10:03:24.97-0700 [TASK/3d855303-24ff-4f0b-9c28-12b56050e97d] OUT /bin/sh: 1: smoggie15: not found
   2020-09-08T10:03:26.16-0700 [TASK/84366dcb-5370-4c52-b754-2e15b9e98779] OUT /bin/sh: 1: smoggie18: not found
   2020-09-08T10:03:28.17-0700 [TASK/28323603-3005-4524-aeba-d9d2b661955c] OUT /bin/sh: 1: smoggie20: not found

After five minutes, I don't see messages for runs 13, 16, 17, or 19

Let's try a command that should succeed: In terminal 1:

for x in {10..20} ; do echo $x; cf run-task node-app1 -c "echo test $x"; done

In terminal 2:

   2020-09-08T10:05:53.64-0700 [TASK/891a5685-910d-4fc8-8fcb-9561c5fbc8d2] OUT test 12
   2020-09-08T10:05:55.88-0700 [TASK/b17360e9-e16a-4c18-81d5-497d1a8e4242] OUT test 16
   2020-09-08T10:05:55.96-0700 [TASK/067e51c9-d469-44e6-abfc-1ca82a300075] OUT test 17
Benjamintf1 commented 3 years ago

Looking at this issue, it looks like the jobs in the namespace for workloads get deleted by eirini(or something) pretty promptly after being completed. This causes the logs for the job to be deleted as well D: