search

datamade / openness-project-nmid

Money Trail NM - New Mexico In Depth's Campaign Finance Explorer
https://moneytrailnm.com
3 stars 1 forks source link

Debug transaction data #182

Closed hancush closed 7 months ago

hancush commented 8 months ago

The transaction import for data from the new SoS site does not appear to be working correctly (e.g., thousands of missing donations). Identify and fix the issue.

hancush commented 7 months ago

Bit more info:

Many transactions appear to be dropped from the data during the new import. I'm sure this is a logical error in the new import script and/or an issue with the way we're filtering transactions in the view code, but I haven't been able to identify where it's happening. Debug import_api_data, in order to ingest the missing transactions.

fgregg commented 7 months ago

the issue seems to be that we delete transactions that match filings as we are importing transactions.

https://github.com/datamade/openness-project-nmid/blob/86a3cbf86b2122f94575d6624afea63f37428121/camp_fin/management/commands/import_api_data.py#L169-L173

it seems likely that there is some uniqueness assumption about filing that is not being satisfied.