Previous log are not display during execution

[tchiotludo]

Seems that the #3022 is back again!

When an execution is started, and output lost of log in realtime, when you click on the gantt charts on a running execution, you will see only the logs in realtime and not the previous one.

id: python-logs
namespace: dev.yann
tasks:
  - id: hello
    type: io.kestra.plugin.scripts.python.Script
    runner: PROCESS
    script: |
      import time
      for i in range(100):
        print(i)
        time.sleep(1) 

Clicking after few seconds will display : 5 , 6, 7, and not the 1, 2, 3, 4

[anna-geller]

on EE there is also an issue that the UI doesn't show all logs (they are cut at a certain limit), see https://preview-ee-kafka.kestra.io/ui/anna/executions/test/python/4z80tyGbgiylub4ngU5Nah/logs?level=TRACE&page=1

repro:

id: python
namespace: test

tasks:
  - id: python
    type: io.kestra.plugin.scripts.python.Script
    taskRunner:
      type: io.kestra.plugin.core.runner.Process
    beforeCommands:
      - pip install requests
    script: | 
      import requests

      response = requests.get('https://api.github.com')
      print(response.status_code)

when you download the log you will see there are more logs captured by the backend than the UI is showing:

2024-07-01T21:05:33.488Z INFO Provided 0 input(s).
2024-07-01T21:05:33.559Z INFO Provided 1 input(s).
2024-07-01T21:05:33.683Z DEBUG Starting command with pid 9097 [/bin/sh -c set -e
pip install requests
python /tmp/2cpot3DpAi5KjB3ZjztdHs/9987286607004781761.py]
2024-07-01T21:05:37.148Z INFO Defaulting to user installation because normal site-packages is not writeable
2024-07-01T21:05:37.512Z INFO Collecting requests
2024-07-01T21:05:37.592Z INFO   Downloading requests-2.32.3-py3-none-any.whl (64 kB)
2024-07-01T21:05:37.615Z INFO      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.9/64.9 KB 2.9 MB/s eta 0:00:00
2024-07-01T21:05:37.707Z INFO Collecting certifi>=2017.4.17
2024-07-01T21:05:37.723Z INFO   Downloading certifi-2024.6.2-py3-none-any.whl (164 kB)
2024-07-01T21:05:37.754Z INFO      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 164.4/164.4 KB 12.2 MB/s eta 0:00:00
2024-07-01T21:05:37.805Z INFO Collecting idna<4,>=2.5
2024-07-01T21:05:37.817Z INFO   Downloading idna-3.7-py3-none-any.whl (66 kB)
2024-07-01T21:05:37.835Z INFO      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 66.8/66.8 KB 6.9 MB/s eta 0:00:00
2024-07-01T21:05:38.073Z INFO Collecting charset-normalizer<4,>=2
2024-07-01T21:05:38.087Z INFO   Downloading charset_normalizer-3.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB)
2024-07-01T21:05:38.103Z INFO      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 142.1/142.1 KB 17.9 MB/s eta 0:00:00
2024-07-01T21:05:38.175Z INFO Collecting urllib3<3,>=1.21.1
2024-07-01T21:05:38.184Z INFO   Downloading urllib3-2.2.2-py3-none-any.whl (121 kB)
2024-07-01T21:05:38.200Z INFO      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.4/121.4 KB 11.1 MB/s eta 0:00:00
2024-07-01T21:05:38.246Z INFO Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests
2024-07-01T21:05:38.479Z WARN   WARNING: The script normalizer is installed in '/home/kestra/.local/bin' which is not on PATH.
2024-07-01T21:05:38.481Z WARN   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2024-07-01T21:05:38.554Z INFO Successfully installed certifi-2024.6.2 charset-normalizer-3.3.2 idna-3.7 requests-2.32.3 urllib3-2.2.2
2024-07-01T21:05:39.054Z INFO 200
2024-07-01T21:05:39.094Z DEBUG Command succeed with code 0

[MilosPaunovic]

This seems to be fixed now with https://github.com/kestra-io/kestra/pull/4468, but I might be testing the wrong thing, can you double check @anna-geller?

[anna-geller]

on develop image currently:

id: myflow
namespace: company.team

inputs: 
  - id: nr
    type: INT
    defaults: 50

tasks:
  - id: log
    type: io.kestra.plugin.scripts.python.Script
    script: |
      import time

      logs = int({{ inputs.nr }}) + 1
      for i in range(1, logs):
          print(f"Generating log messages: {i}")
          time.sleep(0.01)

they don't appear in the right order and not all are captured

maybe we should try on a local build

[MilosPaunovic]

I've been testing and re-testing locally with the example flow from your last comment @anna-geller, but can't seem to reproduce the problem.

It might be that develop image wasn't built properly when you were testing it, could you try one more time, as I've ran also the latest image: kestra/kestra:develop-full and it seems to be working as expected.

[anna-geller]

yup I can confirm it works