bmbouter
commented
2 years ago @mdellweg do you have any insight into why this is failing occasionally?
Open bmbouter opened 2 years ago
bmbouter
commented
2 years ago @mdellweg do you have any insight into why this is failing occasionally?
mdellweg
commented
2 years ago This usually comes down to a task cancel request that was unheard by the worker. I do not know the timing window it slips through though. Another idea that did not immediately validate was that the cancel request was swallowed by the retry logic in the downloaders. But on the other hand, this is the only test we have for really canceling a running task. So i would not want to just remove it.
bmbouter
commented
2 years ago This happened again on last night's nightly. It was the S3 runner, it had:
pushd ../pulp-cli
~/work/pulpcore/pulp-cli ~/work/pulpcore/pulpcore
pytest -v -m pulpcore
============================= test session starts ==============================
platform linux -- Python 3.8.12, pytest-7.1.2, pluggy-1.0.0 -- /opt/hostedtoolcache/Python/3.8.12/x64/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/pulpcore/pulp-cli, configfile: pyproject.toml
plugins: pulpcore-3.20.0.dev0, pulp-certguard-1.6.0.dev0, xdist-2.5.0, forked-1.4.0, pulp-smash-1!0.13.0, custom-exit-code-0.3.0
collecting ... collected 532 items / 517 deselected / 15 selected
tests/scripts/pulpcore/test_access_policy.sh::access_policy.sh PASSED [ 6%]
tests/scripts/pulpcore/test_artifact.sh::artifact.sh PASSED [ 13%]
tests/scripts/pulpcore/test_content_guards.sh::content_guards.sh PASSED [ 20%]
tests/scripts/pulpcore/test_group.sh::group.sh PASSED [ 26%]
tests/scripts/pulpcore/test_group_permissions.sh::group_permissions.sh SKIPPED [ 33%]
tests/scripts/pulpcore/test_orphans.sh::orphans.sh PASSED [ 40%]
tests/scripts/pulpcore/test_pulpexporter.sh::pulpexporter.sh PASSED [ 46%]
tests/scripts/pulpcore/test_pulpimporter.sh::pulpimporter.sh PASSED [ 53%]
tests/scripts/pulpcore/test_role.sh::role.sh PASSED [ 60%]
tests/scripts/pulpcore/test_show.sh::show.sh PASSED [ 66%]
tests/scripts/pulpcore/test_signing_service.sh::signing_service.sh SKIPPED [ 73%]
tests/scripts/pulpcore/test_status.sh::status.sh PASSED [ 80%]
tests/scripts/pulpcore/test_task.sh::task.sh FAILED [ 86%]
tests/scripts/pulpcore/test_user.sh::user.sh PASSED [ 93%]
tests/scripts/pulpcore/test_worker.sh::worker.sh PASSED [100%]
=================================== FAILURES ===================================
_____________________________ test script: task.sh _____________________________
Script returned with exit code 1.
----------------------------- Captured stdout call -----------------------------
true
true
{"pulp_href": "/rerouted/djnd/api/v3/tasks/cab8fddb-c477-4735-9940-a646c16d2bd2/", "pulp_created": "2022-06-14T05:17:54.470866Z", "state": "completed", "name": "pulpcore.app.tasks.orphan.orphan_cleanup", "logging_cid": "082a8726fbcd4ce59352dd76a65f4095", "started_at": "2022-06-14T05:17:54.502746Z", "finished_at": "2022-06-14T05:17:54.830801Z", "error": null, "worker": "/rerouted/djnd/api/v3/workers/e59a3ba6-1744-4f80-b206-7459b9886ce1/", "parent_task": null, "child_tasks": [], "task_group": null, "progress_reports": [{"message": "Clean up orphan Content", "code": "clean-up.content", "state": "completed", "total": 13, "done": 13, "suffix": null}, {"message": "Clean up orphan Artifacts", "code": "clean-up.content", "state": "completed", "total": 13, "done": 13, "suffix": null}], "created_resources": [], "reserved_resources_record": []}
----------------------------- Captured stderr call -----------------------------
SUCCESS [pulp task list --name pulp_file.app.tasks.synchronizing.synchronize --state canceled]
SUCCESS [pulp file remote create --name cli_test_file_remote --url http://pulp-fixtures:8080/file/PULP_MANIFEST]
SUCCESS [pulp file remote create --name cli_test_file_large_remote --url http://pulp-fixtures:8080/file-large/PULP_MANIFEST]
SUCCESS [pulp file repository create --name cli_test_file_repository --remote cli_test_file_large_remote]
SUCCESS [pulp --background file repository sync --name cli_test_file_repository]
SUCCESS [pulp task cancel --href /rerouted/djnd/api/v3/tasks/9f5e1475-a1fd-4aeb-a37b-6b5506be5a68/]
SUCCESS [pulp task list --name pulp_file.app.tasks.synchronizing.synchronize --state canceled]
FAILURE [test 1 -eq 2]
=== STDOUT ===
=== STDERR ===
==============
Started background task /rerouted/djnd/api/v3/tasks/c622a9ae-4772-4973-b1b3-5da8dfb2fb9b/
Done.
Started background task /rerouted/djnd/api/v3/tasks/4aaadbba-70f1-4090-ba17-671387062503/
Done.
Started background task /rerouted/djnd/api/v3/tasks/caeff102-542e-440e-ac09-ece35b7b59c6/
Done.
Started background task /rerouted/djnd/api/v3/tasks/cab8fddb-c477-4735-9940-a646c16d2bd2/
.Done.
=========================== short test summary info ============================
FAILED tests/scripts/pulpcore/test_task.sh::task.sh
===== 1 failed, 12 passed, 2 skipped, 517 deselected in 149.02s (0:02:29) ======
Error: Process completed with exit code 1.
In this case it was in the
streamrunner, but I think it can happen anywhere? I don't know how to reproduce it.