Scheduling Processing Time

[dwillis]

From the April 27 presentation, the issue of when the bulk data is processed and updated was raised. Personally, I feel very strongly that the Library should not sacrifice consistency between the bulk data and Congress.gov in order to preserve timeliness. My preference would be to have the data processed more than once a day, and if 4 a.m. is one of the times, then at least one of the other times should be sometime before noon (8 am or 11 am).

I raised the original issue of moving the processing from 4 am because there were discrepancies between the data and Congress.gov as a result, and feel that there must be a balance between timeliness and consistency, not favoring timeliness over consistency.

[jmarks1992]

Derek's suggestion of updating the bulk data multiple times a day would work for us here at Quorum -- we would then be able to push email updates in the morning based on the 4am update and run a second update later in the morning (post 8am) to make sure that everything syncs with Congress.gov.

At what point does running multiple updates in a day become redundant? Would additional updates beyond the 8am update be beneficial?

[JoshData]

Agreed- Two updates would be a big help. Not having the previous day's activity until late morning the next day (in a one-8am-update schedule) is a very big step backward from THOMAS.

[llaplant]

We are looking at multiple times a day with the first call at 6:00 AM because Congress.gov syncs with key data sources around 5:00 AM.

[llaplant]

We are now running the 114 bill status update job every 4 hours at 4AM, 8AM, 12PM, 4PM, 8PM, and 12AM.

[jmarks1992]

Wonderful! Thank you @llaplant that sounds like it resolves everyone's concerns.

[dwillis]

Agreed, thank you!