CouncilDataProject / cdp-backend

Data storage utilities and processing pipelines used by CDP instances.
https://councildataproject.org/cdp-backend
Mozilla Public License 2.0
22 stars 26 forks source link

bugfix/dont-fail-pipeline-on-bad-picture-urls #147

Closed evamaxfield closed 2 years ago

evamaxfield commented 2 years ago

Description of Changes

Include a description of the proposed changes.

This PR fixes or adds a feature (depending on how you look at it). We are experiencing some failures on the King County reprocessing: https://github.com/CouncilDataProject/king-county/runs/4729015978?check_suite_focus=true

This is due to the seat images failing to download, not because they have moved or don't exist but because our request has been denied by the host. This change makes it so that the pipeline will simply log the error and continue the rest of the pipeline. This imo is completely fine as seat and person images are optional anyway.

cc @dphoria because we probably want to update the king county seat images :joy:

codecov[bot] commented 2 years ago

Codecov Report

Merging #147 (d0f273d) into main (c25dd22) will increase coverage by 0.00%. The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #147   +/-   ##
=======================================
  Coverage   94.82%   94.82%           
=======================================
  Files          50       50           
  Lines        2531     2532    +1     
=======================================
+ Hits         2400     2401    +1     
  Misses        131      131           
Impacted Files Coverage Δ
cdp_backend/pipeline/event_gather_pipeline.py 85.67% <100.00%> (+0.03%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update c25dd22...d0f273d. Read the comment docs.

dphoria commented 2 years ago

cc @dphoria because we probably want to update the king county seat images joy

Got it!