drop and reimplement io_in_bg

mbukatov commented 2 years ago

Issues with current io_in_bg design:

workload is running in a thread of ocs-ci pytest run
log pollution (log messages of io_in_bg ends up scattered in logs of test cases)
prone to errors
increased cost of ci analysis

Proposed solution:

don't use threads to run the background io
remove all bg io code from ocs-ci
implement the bg io via fio container (with configuration provided via a config map) k8s deployment, which could be scaled up and down to control it's run

To be considered:

only test cases which will indicate that they can handle background io would be considered for bg io run

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 30 days if no further activity occurs.

mbukatov commented 2 years ago

I"m running into this every CI run, should not be flagged as stale, unless ~io_in_bf~ is dropped.

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 30 days if no further activity occurs.

github-actions[bot] commented 1 year ago

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

github-actions[bot] commented 1 year ago

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

github-actions[bot] commented 1 year ago

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

ebenahar commented 1 year ago

@Akarsha-rai @mbukatov as of now, the logging of IO in BG is quite minimal. Should we still consider disabling it for the general use case?

mbukatov commented 1 year ago

I still believe it's worth to convert it to a deployment or a job, and then run it in selected test runs only while giving the tests a way to disable it temporarily if necessary.

This would address at least the interference issue, I usually see few cases of tests failed on stopping IO in BG.

ebenahar commented 1 year ago

Converting to a job would indeed improve the stabilization. We can keep this issue to track such conversion.

As for the option to pause the IO exists here - https://github.com/red-hat-storage/ocs-ci/blob/49d8a77774cc8568714d7502051e846f9ef247fe/tests/conftest.py#L1712

Here is a reference for its usage - https://github.com/red-hat-storage/ocs-ci/blob/3b04171d5cfaea8921419164f96517a098201354/tests/ecosystem/upgrade/test_resources.py#L51

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 30 days if no further activity occurs.

github-actions[bot] commented 1 year ago

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

github-actions[bot] commented 1 year ago

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

github-actions[bot] commented 1 year ago

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

DanielOsypenko commented 11 months ago

https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#ocs/launches/493/12999/595182/595183/595185/log?logParams=history%3D595185%26page.page%3D1

DanielOsypenko commented 9 months ago

taking this for implementation

github-actions[bot] commented 6 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 30 days if no further activity occurs.

DanielOsypenko commented 5 months ago

test_monitoring_reporting_ok_when_idle

ALL THE FAILED TESTS WERE RUNNING ON EXTERNAL MODE CLUSTER

up to 10% of tests running on ODF 4.13 Fail with:

Message: failed on setup with "Exception: io_in_bf failed to stop after 600 timeout, bug in io_in_bf (of ocs-ci) prevents execution of test cases which uses this fixture, rerun the affected test cases in a dedicated run and consider ocs-ci fix" Type: None Text: measurement_dir = '/tmp/pytest-of-jenkins/pytest-1/measurement_results' threading_lock = <unlocked _thread.lock object at 0x7fb3ebd763c0>

reason - we need to turn On ceph metrics on external clsuter before test run PR #8457

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 30 days if no further activity occurs.

github-actions[bot] commented 3 weeks ago

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

red-hat-storage / ocs-ci

drop and reimplement io_in_bg #5177