When we are storing links to the matter attachments during the event ingestion pipeline store_event_processing_result task, we run existance checks on every supporting file. Because there can be a lot of these supporting files, we occasionally encounter request rejections for hitting the host too often / too quickly.
Expected Behavior
What did you expect to happen instead?
The file should be skipped. Simply catching the exception and moving on instead of failing the pipeline.
Reproduction
Steps to reproduce the behavior and/or a minimal example that exhibits the behavior.
Describe the Bug
A clear and concise description of the bug.
When we are storing links to the matter attachments during the event ingestion pipeline
store_event_processing_resulttask, we run existance checks on every supporting file. Because there can be a lot of these supporting files, we occasionally encounter request rejections for hitting the host too often / too quickly.Expected Behavior
What did you expect to happen instead?
The file should be skipped. Simply catching the exception and moving on instead of failing the pipeline.
Reproduction
Steps to reproduce the behavior and/or a minimal example that exhibits the behavior.
https://github.com/CouncilDataProject/seattle-staging/runs/4930903577?check_suite_focus=true
Environment
Any additional information about your environment.
Ubuntu 20.04 and cdp-backend v3.0.3 -- see GitHub Actions details.