Ingestion of images stops unexpectedly after cleanup

[dhruvkb]

Description

All CI flakiness has boiled down to the ingestion of images stopping unexpectedly after the cleanup step. Rerunning ingestion almost surely fixes the issue.

We should investigate the issue and find out the cause. If it is incurable, we should add a step to retry the ingestion which is almost guaranteed to succeed the second time.

This also causes WordPress/openverse-api#562.

Reproduction

See failed runs:

• https://github.com/WordPress/openverse-api/runs/6527207725?check_suite_focus=true#step:8:702
• https://github.com/WordPress/openverse-api/runs/6527646327?check_suite_focus=true#step:11:480

Resolution

• [ ] 🙋 I would be interested in resolving this bug.

[dhruvkb]

WordPress/openverse-api#802 should reduce the flakiness in our CI by retrying ingestion if it unexpectedly stops but the real cause and the proper solution for this issue are yet to be found.

[dhruvkb]

^ Accidental click.

[zackkrida]

@WordPress/openverse-api have we observed this behavior in the API CI still?

[dhruvkb]

We covered over this problem by repeating the process if it fails (up to 3 retries) so this problem doesn't appear in CI anymore. But the underlying cause has not been fixed. @obulat's PR WordPress/openverse-api#839 removes the problematic step and that will fix the problem at the cause.

[sarayourfriend]

@dhruvkb Do we still need this issue? Should it be re-written as an issue to remove the cleanup steps or (not sure of context) folded into a side effect of the data normalisation project (cc @obulat)?