CouncilDataProject / cdp-backend

Data storage utilities and processing pipelines used by CDP instances.
https://councildataproject.org/cdp-backend
Mozilla Public License 2.0
22 stars 26 forks source link

feature/reduce-event-gather-complexity #232

Closed evamaxfield closed 1 year ago

evamaxfield commented 1 year ago

Had parts of this code sitting around. Finished it up in ~30 minutes. Need to test on a dev instance but pretty confident it will work.

Main change is resolving #230 -- moving from processing all sessions in a single flow to producing multiple flows, one for each session. I haven't touched any multiprocessing just yet to be safe.

evamaxfield commented 1 year ago

Heading out for the night, might get a chance to test this tomorrow. If it works on a dev instance will fast merge and deploy

codecov[bot] commented 1 year ago

Codecov Report

Merging #232 (3169ec3) into main (ad09d9c) will decrease coverage by 0.39%. The diff coverage is 39.53%.

:exclamation: Current head 3169ec3 differs from pull request most recent head 95a29a7. Consider uploading reports for the commit 95a29a7 to get more accurate results

@@            Coverage Diff             @@
##             main     #232      +/-   ##
==========================================
- Coverage   72.01%   71.62%   -0.39%     
==========================================
  Files          50       50              
  Lines        3312     3330      +18     
==========================================
  Hits         2385     2385              
- Misses        927      945      +18     
Impacted Files Coverage Δ
cdp_backend/bin/process_special_event.py 0.00% <0.00%> (ø)
cdp_backend/bin/run_cdp_event_gather.py 0.00% <0.00%> (ø)
cdp_backend/tests/utils/test_file_utils.py 100.00% <ø> (ø)
cdp_backend/pipeline/event_gather_pipeline.py 82.20% <86.66%> (+0.05%) :arrow_up:
...ckend/tests/pipeline/test_event_gather_pipeline.py 98.36% <100.00%> (+0.01%) :arrow_up:
cdp_backend/utils/file_utils.py 88.29% <100.00%> (ø)