Making "updated" fields available & searchable in API endpoints

[llienfec]

RAD's contact portal uses workflows in ServiceNow to pull info from 4 API endpoints (below). Currently, it's pulling 100% of the data in each endpoint every night. To optimize the time it takes to pull data and to reduce server load, we are looking for fields to use as "updated" fields in the data, surfacing those in the API, and making the fields searchable. Approach looks possible after borrowing @lbeaufort 's scoping and research expertise.

Name	Endpoint
FEC Candidate Information	https://api.open.fec.gov/v1/committee/${committee_id}/candidates
FEC Committee By ID	https://api.open.fec.gov/v1/committee/${committee_id}
FEC Committee Information	https://api.open.fec.gov/v1/committees
FEC RAD Analyst	https://api.open.fec.gov/v1/rad-analyst

Possible "updated" fields

/committees/

• last_filed_f1 is in API results.
• [ ] Need to add search capability. WIP PR: https://github.com/fecgov/openFEC/pull/3316

/committee/${committee_id}/candidates

resource: candidates.CandidateView

• last_filed_f2 field is in API results. Not preferable to use b/c committees often do not file F2s (they can just not respond to disavowal notices)
• has load_date field and needs research to make sure this is the updated date
• [ ] need to be able to search by load_date

/rad-analyst/

• need to surface the date that the analyst assigned to a particular committee
• no field in current API results
• data has a pg_update field; doing research to see if this field will work
• [ ] update the view rad_cmte_analyst_search_vw
• [ ] bring the field into the API output
• [ ] add a filter for pg_update that field to API

If pg_update field works - we may want to see if we can use this for all of the "updated" fields for consistency.

• [ ] Call this load_date? Look at load_date and other last_update_dt type fields in the API

Future work/best practices

We may want to consider making sure that an "updated" field is available and searchable in each API endpoint as a best practice. This would help users who regularly pull our data pull over most recent updates instead of calling all entries.

[llienfec]

Talked to the ITC contractors about a question we had re: workflow.

• Confirmed that updating /committees/ will take care of timing issues with both pulling the committee info and add'l committee info.
• Asked if it also takes care of the candidate info, which is pulled using the committee ID. Said no b/c it wouldn't capture cases where candidate info is updated but committee information is not.

I asked them to research if/how we use the info from this endpoint, so we can prioritize optimizing this endpoint, not pulling over the database anymore, or something in between.

[llienfec]

Talked to the contractors about the endpoint they're using to pull candidate info: https://api.open.fec.gov/v1/committee/${committee_id}/candidates

It doesn't look like we're using the info from this endpoint in ServiceNow. For now, we don't need to work on optimizing this endpoint and I'm going to suggest that they stop calling the API for this info. If we (RAD or another program office) need this information in the future, I'm going to suggest that they switch endpoints to the /candidates/ endpoint. This endpoint has a few fields that we could pick from for the updated date including:

• last_f2_date
• last_filed_date
• load_date

[lbeaufort]

@llienfec thanks for the info! To make sure I understand, I only need to make these changes?

/committees/ last_filed_f1 is in API results.

• [ ] Need to add search capability. WIP PR: #3316

/rad-analyst/ need to surface the date that the analyst assigned to a particular committee no field in current API results data has a pg_update field; doing research to see if this field will work

• [ ] update the view rad_cmte_analyst_search_vw
• [ ] bring the field into the API output
• [ ] add a filter for pg_update that field to API

[llienfec]

Yep, only those changes. Thanks so much for all of your work on this!