integration tests not running in stage

[codificat]

Describe the bug

The integration tests in the stage environment have not completed during the past week.

To Reproduce Steps to reproduce the behavior:

1. Look at the integration-tests CronJob in thoth-infra-stage namespace in the stage cluster
2. check the list of Jobs ran
3. Notice that the completions is 0/1 for the recent Job runs

Expected behavior

Integration tests should run.

Additional context

Looking at yesterday's Job run, its pod (integration-tests-27564480-lntgz) is marked as Error, but a look at the logs does not show an obvious error to me. This is the full log output from the pod:

/opt/app-root/lib64/python3.8/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
--------------------------------------------------------------------------------
> Tests are executed against ocp4-stage deployment
> Tests are executed against User API at stage.thoth-station.ninja
> Tests are executed against Management API at management.stage.thoth-station.ninja
> Tests are executed against Amun API at amun.stage.thoth-station.ninja
--------------------------------------------------------------------------------
Tests are executed using ['--show-timings', '-f', 'html', '-o', 'behave-report.html', '--tags', '~@seizes_middletier_namespace']

[codificat]

/priority critical-urgent /assign @harshad16

[harshad16]

/triage accepted

[harshad16]

/sig devsecops

The issue was due to the excessive request for data ingestion. caused the investigator(Kafka consumer) to halt the process without halting the consumer topic. which results in delay, it functions back once the halt process of the investigator is back. In this process the halt period topics are missed, this is one a blue moon situation we would wait for the next appearance of this issue. To gather information for investigation as of now we lack information for further investigation.

closing this issue as the current integration test is back and working.