fedspendingtransparency / usaspending-api

Server application to serve U.S. federal spending data via a RESTful API
https://www.usaspending.gov
Creative Commons Zero v1.0 Universal
309 stars 112 forks source link

[Hotfix] [DEV-10961][QAT] Update the COVID19-FABA Elasticsearch index to be un-nested #4094

Closed aguest-kc closed 5 months ago

aguest-kc commented 6 months ago

Description: Our COVID19-FABA index in Elasticsearch has a large number of nested objects and recently we ran into an issue where one Award (and it's FABA records) are too large to POST to Elasticsearch. This means that we cannot update our COVID FABA data without changing the structure of the index. This will update the index's structure so that the FABA records related to an Award are no longer nested objects within that Award, reducing the data that is sent in a single POST request.

Technical details: Our COVID19-FABA index in Elasticsearch has a large number of nested objects and recently we ran into an issue where one Award (and it's FABA records) are too large to POST to Elasticsearch. This means that we cannot update our COVID FABA data without changing the structure of the index. This will update the index's structure so that the FABA records related to an Award are no longer nested objects within that Award, reducing the data that is sent in a single POST request.

Requirements for PR merge:

  1. [x] Unit & integration tests updated
  2. [ ] Necessary PR reviewers:
    • [ ] Backend
  3. [x] Data validation completed
  4. [x] Appropriate Operations ticket(s) created
  5. [x] Jira Ticket DEV-10961:
    • [x] Link to this Pull-Request
    • [x] Performance evaluation of affected (API | Script | Download)
    • [x] Before / After data comparison

Area for explaining above N/A when needed:

2. API documentation updated
No API documentation needs to be updated.

4. Matview impact assessment completed
No matviews are affected by this change.

5. Frontend impact assessment completed
The frontend is not affected by this change.