Open JenySadadia opened 2 months ago
Some comments based on an initial investigation:
Logger
configurations to dump logs to file instead of stdout
here https://github.com/kernelci/kernelci-pipeline/blob/main/src/logger.py#L26docker logs
there. See https://dozzle.dev/guide/log-files-on-disk#following-log-files-on-diskDozzle
and look for another alternative that works well with log files.Finally i found approach that helped me in past. In general terms it is described here: https://docs.python.org/3/howto/logging-cookbook.html
You can add multiple handlers with different log levels to Logging, so this way you can save in file verbose submission info, and leave for stdout (dozzle/kubernetes) generic info, simultaneously.
On the staging instance, we are running Dozzle container for publishing API and Pipeline service logs. i.e. https://staging.kernelci.org:9088/
One problem with it is whenever services restart, we lose all the previous logs and hence also lose a trail of failures as well. It's not possible to catch all failures without log dump. Hence, enable a mechanism to dump logs to log files to debug the services at a later time.