fecgov / openFEC

The first RESTful API for the Federal Election Commission. We're aiming to make campaign finance more accessible for journalists, academics, developers, and other transparency seekers.
https://api.open.fec.gov/developers
Other
483 stars 106 forks source link

Discrepancy between Bulk Data files and committee top-lines #5911

Closed mchonofsky closed 3 months ago

mchonofsky commented 4 months ago

What we’re after

The sum of items in the 'oppexp' bulk data files do not match the operating expenditures in the committee master. Is it possible to get the numbers to match? Not sure if this is exactly the right place for this issue but I hope you can point me in the right direction. Thank you!

References/resources/technical considerations

For one committee that I have checked:

The discrepancy also exists in other cycles, committees, and lines on the committee page. It's not consistently greater than or less than zero.

PaulClark2 commented 4 months ago

@mchonofsky The operating expenditure file includes memo items. You'll need to exclude these transactions for the total to match what's displayed on the financial summary page of the committee's profile page. See Form 3X and its instructions for information about memo items.

MEMO_CD SUM(TRANSACTION_AMT)
$654,060.84
X $49,967.42
mchonofsky commented 4 months ago

@PaulClark2 that's great, very helpful, thank you! Do you know what accounts for the remaining discrepancy? I looked at another example with only a handful of disbursements, where the total was likewise somewhat less than the financial summary value, and I was able to account for every line in the bulk data matching an entry on the disbursements search page. So I'm not sure what the source of the missing value might be. Do I need to parse the .fec files directly?

image

PaulClark2 commented 4 months ago

@mchonofsky the discrepancies you are seeing are because not all the Q2/M7 reports have been processed. The financial summary page is usually updated overnight. If a filing comes in after 7:30 pm, it takes an additional day for the financial summary data to be updated. However, the transaction-level data takes up to 30 days to process with most being processed within 15 days of receipt.

The operations log endpoint in our openFEC API will allow you to determine if a report's transactions have been processed. filings --> /v1/operations-log/

mchonofsky commented 4 months ago

Okay, that makes sense, but I'm still a little confused because this is a 2022 report. Are you suggesting that C00708552 has submitted a Q2/M7 report for 2022 recently or included 2022 expenses on its most recent report? As another test case, C00549014 has a Statement of Termination in 2023 but the summary page and the sum of Operational Expenditures in the 2024 cycle don't appear to match either. I hope I'm not missing something obvious and I'm really grateful for your help!

PaulClark2 commented 4 months ago

I forgot you are looking at 2022 data. I'll look at your specific examples a little later today.

PaulClark2 commented 3 months ago

@mchonofsky I needed to confirm some operating expenditure reporting requirements. There is no expectation that the sum of the itemized operating expenditure transactions will equal the amount reported on Line 21b of the Detailed Summary Page of a report.

Expenditures that aggregate to less than $200 in a calendar year to a single vendor are not itemized. So, they will not be in the bulk data file, but the amount will be included in the amount report on Line 21b of the Detailed Summary Page.

Separate Segregated Funds (SSF) generally do not report operating expenditures because they covered by the connected organization.

mchonofsky commented 3 months ago

Got it. Thank you for helping me understand!

PaulClark2 commented 3 months ago

You're welcome.