Research making totals-level data available for filings after 7:30 pm using data from real_efile

[fecjjeng]

This is a followup ticket from #3701: "Research making totals-level data available for filings after 7:30 pm"

[fecjjeng]

We had considered several possible solution to solve problem mentioned in #3701. Solution #3, get data on regular financial reports (F3, F3P, and F3X) from real_efile would be a possible solution. But will requires major lifting, involved changes on fundamental process that we currently relying on. Also, it is still depending on the time required for data feed from committees and Salient, our contractor company who process data from committees. Recommended to re-consider after 2020 election.

[fecjjeng]

Ideally for all financial forms but just summary data (pass 1 data), for regular financial reports (F3, F3P, and F3X), real_efile only, not real_pfile.

Some query examples provided by @PaulClark2. Recorded here for future reference: select f3.comid, F3.REPID, f3.rptcode, f3.through_date, (case when vs.orig_sub_id is not null then 'X' else null end) most_recent_filing_flag, f3.processed_flag from (select ef3.comid, eF3.REPID, ef3.RPTCODE, ef3.THROUGH_DATE, (case when procF3.sub_id is not null then 'X' else null end) processed_flag from REAL_EFILE.F3 ef3 left join disclosure.nml_form_3 procF3 on procF3.file_num = ef3.repid) f3 left join disclosure.v_sum_and_det_sum_report vs on F3.REPID = vs.file_num where f3.comid = 'C00692848' order by 4;

select f3x.comid, F3x.REPID, f3x.rptcode, f3x.through_date, (case when vs.orig_sub_id is not null then 'X' else null end) most_recent_filing_flag, f3x.processed_flag from (select ef3x.comid, eF3x.REPID, ef3x.RPTCODE, ef3x.THROUGH_DATE, (case when procF3x.sub_id is not null then 'X' else null end) processed_flag from REAL_EFILE.F3x ef3x left join disclosure.nml_form_3x procF3x on procF3x.file_num = ef3x.repid) f3x left join disclosure.v_sum_and_det_sum_report vs on F3x.REPID = vs.file_num where f3x.comid = 'C00690362' order by 4;

[PaulClark2]

Just a note about my examples, I looked in reps for committees that filed after 7:30 pm to ensure we'd see processed and raw filings in the query.

[lbeaufort]

Relevant issue about filings submitted after 7:30pm not appearing: https://github.com/fecgov/openFEC/issues/4018

We may want to research the feasibility of making filings available that were filed before the midnight deadline.

[davidnir]

Hi. Just wondering if there were any updates on this issue. Thank you!

[PaulClark2]

@davidnir we are still doing internal testing. Our tests are focused on making sure all of our processes that update data before start of business Eastern time. We'll update this issue once we've completed our testing.

Currently the filings are available but not incorporated with the financial summaries. You can find the filers in these places:

• https://www.fec.gov/data/filings/?data_type=efiling
• https://www.fec.gov/fec-electronic-filing-report-retrieval/
• https://api.open.fec.gov/developers/#/efiling

[davidnir]

Thanks for the reply!

[davidnir]

Hi! Any further updates on this? Thank you!

[david-nir]

Hi, @PaulClark2. I'm wondering if there have been any further updates on this. Thank you!

[PaulClark2]

We do not currently have the resources to do the work required to do this work.

[david-nir]

I appreciate the quick reply. I am sorry to hear that. I am curious, though, about this suggestion in particular:

One idea is to run the b_nightly process again after midnight - the RAD modues will be updated much later, but the data wouldn't be any more stale than it is currently.

Is there any chance that this might be feasible?