thoth-station / kebechet

I'm Kebechet bot, goddess of freshness - I will keep your source code fresh and up-to-date
24 stars 20 forks source link

some workflow seems to fail #1147

Closed goern closed 2 years ago

goern commented 2 years ago

Bug description

2022-09-26 08:15:31,014 1 WARNING thoth.common:340: Logging to a Sentry instance is turned off
2022-09-26 08:15:31,423 1 INFO thoth.run_kebechet_administrator:52: Thoth workflow-helpers task: run_kebechet_administrator v0.9.13+thamos.1.27.11.common.0.36.4.python.0.16.10analyzer.0.1.8.storages.0.73.3
2022-09-26 08:15:32,122 1 INFO thoth.run_kebechet_administrator:201: Kebechet administrator triggered by: thoth.solver.solved-package
2022-09-26 08:15:33,128 1 INFO thoth.run_kebechet_administrator:205: Number of messages to be sent: 0
2022-09-26 08:15:33,128 1 ERROR thoth.workflow_helpers.common:81: [Errno 2] No such file or directory: '/mnt/workdir/messages_to_be_sent.json'
Traceback (most recent call last):
File "/opt/app-root/src/thoth/workflow_helpers/common.py", line 74, in store_messages
with open(MSG_OUT_FILE, "r") as json_file:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/workdir/messages_to_be_sent.json'

Steps to Reproduce

Steps to reproduce the behavior:

  1. Go to https://console-openshift-console.apps.ocp4.prod.psi.redhat.com/k8s/ns/thoth-backend-stage/pods
  2. check the log of the main container of one of the kebechet-admin pods
  3. See error

Actual behavior

n/a

Expected behavior

n/a

Additional context

n/a

/kind bug /priority critical-urgent /assign @KPostOffice /assign @VannTen

VannTen commented 2 years ago

From the log of the wait containers in those pods and the annotations it looks like this messages_to_be_sent.json is the output of that pod.

Together with the fact that the number of messages to be sent is 0 maybe the process tried to reread the file it writes, and fail when it did not write it because there was no messages ?

I'll take a look.

VannTen commented 2 years ago

Hum, it's not as bad as it looks. We just log it wrong :

    # Store message to file that need to be sent.
    try:
        with open(MSG_OUT_FILE, "r") as json_file:
            if os.stat(MSG_OUT_FILE).st_size != 0:
                all_messages: list = json.load(json_file)
                if type(all_messages) != list:
                    raise TypeError(f"Message file must be a list of messages. Got type {type(all_messages)}")
                all_messages = all_messages + output_messages
    except Exception as e:
        _LOGGER.exception(e)
        all_messages = output_messages

_LOGGER.exception is _LOGGER.error with extra stuff. This should probably be more specific and not and error.