search

WordPress / openverse

Openverse is a search engine for openly-licensed media. This monorepo includes all application code.
https://openverse.org
MIT License
240 stars 194 forks source link

Ingestion of images stops unexpectedly after cleanup #699

Open dhruvkb opened 2 years ago

dhruvkb commented 2 years ago

Description

All CI flakiness has boiled down to the ingestion of images stopping unexpectedly after the cleanup step. Rerunning ingestion almost surely fixes the issue.

We should investigate the issue and find out the cause. If it is incurable, we should add a step to retry the ingestion which is almost guaranteed to succeed the second time.

This also causes WordPress/openverse-api#562.

Reproduction

See failed runs:

Resolution

dhruvkb commented 2 years ago

WordPress/openverse-api#802 should reduce the flakiness in our CI by retrying ingestion if it unexpectedly stops but the real cause and the proper solution for this issue are yet to be found.

dhruvkb commented 2 years ago

^ Accidental click.

zackkrida commented 2 years ago

@WordPress/openverse-api have we observed this behavior in the API CI still?

dhruvkb commented 2 years ago

We covered over this problem by repeating the process if it fails (up to 3 retries) so this problem doesn't appear in CI anymore. But the underlying cause has not been fixed. @obulat's PR WordPress/openverse-api#839 removes the problematic step and that will fix the problem at the cause.

sarayourfriend commented 1 year ago

@dhruvkb Do we still need this issue? Should it be re-written as an issue to remove the cleanup steps or (not sure of context) folded into a side effect of the data normalisation project (cc @obulat)?