[30k]: Search API

[widal001]

Key links

• Deliverable specification
• Simpler Grants.gov full roadmap
• 10k milestones

Description

• What: Publicly launch the API and communicate it to our stakeholders.
• Why: Enables us to begin collecting feedback from system-to-system users (S2S) and planning new API features.
• Who
  • Federal S2S users
  • External S2S users

[!NOTE] For more information about this deliverable, please refer to the deliverable specification linked above.

[EOKENAVA]

Weekly Update

What has been done?

• Strategy session with HHS and Nava

Status On Track

Risk None

Deliverable Date May 14th

What will be done?

• Access to codebase
• Move forward with the Search work using Michael's defined approach
• Run the logic of the data in prod
• Start SIA for the non-public, PII, and other data that we'll need in the next 12 month roadmap

[lucasmbrown-usds]

Could we clarify what this update means: "Run the logic of the data in prod"? Thanks. I'm guessing it's basically, compare opportunity listings we have in grants.gov PROD to the opportunity listings we have in simpler.grants.gov PROD, and make sure they're the same which will confirm the logic in Simpler. But I'm not 100% sure if that's what it means?

[chouinar]

@lucasmbrown-usds I think that is what it was referring to - just the additional set of data validations to confirm our understanding of the data as what we have access to in dev right now might not be representative of real data.

[EOKENAVA]

Weekly Update

What has been done? Setup & create tests for the copy-oracle-data command Develop strategy and plan for data modeling Completed research/discussion around data model session with stakeholders like Narmada, Lucas, and Julius

Risk None

Deliverable Date May 14th

What will be done? Adjust of the database schema of the opportunity tables Adjust search API model to use the new schema & fields

[EOKENAVA]

Weekly Update

What has been done?

• Adjusted the database schema of the opportunity tables
• Adjusted the API schema to match the database schema
• Added filtering to the API for all faceted fields (eg. opportunity status, funding instrument, agency, etc.)
• Modify gunicorn config to use recommended number of workers and threads

Risk None

Deliverable Date May 14th

What will be done?

• Implement support in the search endpoint to use the query box
• Updates for query param logging formatting (to be easily parsed by New Relic when implemented)
• Begin script that will populate current opportunity information once imported from Oracle DB

[margaretspring]

Weekly Update

What has been done?

• Added support to order the response by more fields
• Implement support in the search endpoint to use the query box

Risk None

Deliverable Date May 14th

What will be done?

• Search API - API development (50% done) Search API - Expanded data model (90% done - working on last ticket) Search API - Security for additional fields (50% done - identified strategy for SIA and draft due April 2) Search API - ELT Decision (35% done - still working through deskwork and PoCs on options)
• Updates to how our database models are defined to allow for easily setting up tables in more schemas
• Update the validation of the search endpoint to avoid a few misleading/confusing edge cases with empty filters
• Modify the agency filtering logic to being an exact match

[margaretspring]

Weekly Update

• [10k]: API development
  • status 29 points initially estimates, 43 pointed, 42 completed
  • almost completed this body of work
  • completed
    • [Task]: Create backend script that sets the current opportunity summary for each opportunity #1365
    • [Task]: Organize tests to be easier to read / spread across a few more files #1524
    • [Task]: Modify agency filter to do exact matches rather than the strbegins approach we have #1494
    • [Task]: Modify search filter schema to require at least one filter be set for a given field #1493
    • [Task]: Implement query filtering to the search endpoint #1455
• [10k]: Expanded data model
  • status 9 points initially estimates, 17 pointed, 17 completed
  • finished this milestone/10k
  • completed
    • [Task]: Restructure db session setup to allow for easy use of multiple schemas #1270
• • status 60 points initially estimates, 43.2 pointed, 25 completed
    • getting close to choosing a transform tool. significant OOO so work stalled a little
    • completed
  • [Task]: Specify and document a list of evaluation criteria for Transform technology #1299
  • [Task]: Research a shortlist of Transform technology solutions #1305
  • [Task]: Desk research against evaluation criteria: dbt #1306
• [10k] Security for Search API Additional Fields
  • status 58 points initially estimates, 7.1 pointed, 2 completed
  • SIA second draft to be done this upcoming sprint

Status In progress

Risk SIA acceptance for Security / PII & Non-Public data in Simpler.grants.gov (Non-Public Data is critical, PII is nice to have at this point)

Deliverable Date May 14th

What will be done?

• Transformation: Desk research against evaluation criteria: dbt, evaluate pushing from Oracle to Postgres, specify and documenting a list of evaluation criteria for Transform technology, and research a shortlist of Transform technology solutions
• Security: will continue the SIA documentation
• API: some fixes to current db structure but will complete that

[margaretspring]

Weekly Update 4/11

What has been done

• [10k]: API development
  • status 29 points initially estimates, 46 pointed, 43 completed
  • Remaining task is to investigate relevancy
  • completed
    • [Task]: Update opportunity factories to have more realistic data #1543
• [10k]: Expanded data model
  • status 9 points initially estimates, 17 pointed, 17 completed
  • finished this milestone/10k
• • status 60 points initially estimates, 50.1 pointed, 30 completed
    • landed on using python over dbt and sqlmesh, working on an ADR
    • completed
  • [Task]: Build POC: SQLMesh #1310
• [10k] Security for Search API Additional Fields
  • status 58 points initially estimates, 7.1 pointed, 2 completed
  • SIA second draft to be done this upcoming sprint

Status In progress

Risk SIA acceptance for Security / PII & Non-Public data in Simpler.grants.gov (Non-Public Data is critical, PII is nice to have at this point)

Deliverable Date May 14th

What will be done?

• Transformation: ADR and python app structure
• Security: will continue the SIA documentation

[widal001]

Updated ticket format to reduce discrepancies with deliverable spec.

[margaretspring]

Weekly Update 4/17

Milestones: [10k]: API development (90%) | [10k]: Expanded data model (100%) | [10K]: Transformation (50%) | [10k] Security for Search API Additional Fields (80%)

• Work completed outside of transformation largely completed
• Transformation work started as planned in last two sprints of 30K
• Security review meeting 4/17

What has been done?

• Transformation work ticketed and refined
• 2nd SIA draft sent

Status In progress

Risk

• SIA acceptance
• Transformation work (currently on schedule)

Deliverable Date 5/14/24

What will be done?

• Security and transformation work

[margaretspring]

Weekly Update 4/24

What has been done?

• Final draft of Security SIA
• Adjustments to DB schema to support ELT Implementation
• Scaling database in preparation for user testing
• Continued transformation work & processing

Status In Progress

Risk

• If additional work is necessary for approval of pending SIAs, additional work to address has been scheduled and may push us beyond May 14th.
• If approval of pending SIAs is delayed for other reasons, project can't complete the security requirements for this 30K.
• If transformation work encounters unexpected obstacles, completion of DoD for 30k would be at risk.

Deliverable Date 5/14/24

What will be done?

• Continue security & transformation work
• Database expanding data model and scaling capacity.

[lucasmbrown-usds]

Small nitpick here: the comment above says "Final draft of Slack SIA", but I think you actually mean "Final draft of Search SIA". There have been two SIAs floating around recently (Slack and Search).

[acouch]

but I think you actually mean "Final draft of Search SIA".

Yes, thank you.

[margaretspring]

Weekly Update 5/1

What has been done?

• Requested access to new data tables after security approval
• Setup the transformation process
• Expanded the loading process

Status In Progress

Risk

• Finishing transformation work is targeted for the end of the sprint

Deliverable Date 5/14

What will be done?

• The transformation process will be finalized and API published

[sarahknoppA6]

Development team has updated anticipated delivery date to 5/28 given additional work to improve the data model. https://betagrantsgov.slack.com/archives/C06F55BKLKX/p1714684814304889

[acouch]

Weekly Update 5/8

What has been done?

• Loading from legacy to staging database
• Transformation code created for opportunity, summary, listing created but still in development

Status

• In Progress

Risk

• Transformation work

Deliverable Date

• Sprint 24 - May 28

What will be done?

• Finishing last table #1749
• Testing loaded data
• Running load and transform steps in stage