datamade / openness-project-nmid

Money Trail NM - New Mexico In Depth's Campaign Finance Explorer
https://moneytrailnm.com
3 stars 1 forks source link

Use the most recent filing, if more than one "final" filing was submitted #208

Closed hancush closed 1 month ago

hancush commented 1 month ago

Overview

Sometimes filings are marked as final, even though another copy was submitted later. For example:

We could continue to try to keep track of these instances, but that requires manual intervention in a loooong feedback loop. Instead, where there is more than one final filing for a particular period, use the most recently filed one.

I added a log message to indicate when this is happening, as well.

Notes

My manually triggered workflows are running significantly slower than the repo triggered ones, including the failed run that happened overnight (2 it/s vs. > 40 it/s for candidates, PACs, and their filings, despite not touching that code in this PR). Want to let this run via the usual ETL and see what happens.

Testing Instructions

Manually triggered a workflow. It ran until it timed out.