usdigitalresponse / grants-ingest

Ingests and indexes data related to grants
Apache License 2.0
6 stars 3 forks source link

[Forecasted Pipeline] Persist forecasted grants in DynamoDB #861

Open jeffsmohan opened 1 week ago

jeffsmohan commented 1 week ago

Why is this issue important?

We want forecasted grants available in our dataset (alongside active grants) so we can integrate them into the product for our users.

Current State

The PersistGrantsGovXMLDB lambda only handles active grants today, not forecasted grants.

Expected State

We want the PersistGrantsGovXMLDB lambda to also take the XML snippets for forecasted grants (as produced by the change in #330) and persist those into our DynamoDB database alongside active grants.

Implementation Plan

Blocked on #3212 and #3213 Edit (authored by Marissa): This ticket isn't blocked by #3212 or #3213 if flags are used in similar fashion to IS_FORECASTED_GRANTS_ENABLED in this PR. Ideally, the same flag (IS_FORECASTED_GRANTS_ENABLED) would be used.

  1. Update the type opportunity grantsgov.OpportunitySynopsisDetail_1_0 in handler.go with a new type that can handle both grant opportunities and forecasted grants, with a common interface for logging. (The rest of the DynamoDB update should work, as it's agnostic to the shape of the data.)
  2. Create a second lambda_function in terraform/main.tf to invoke the lambda for forecasted grants as well:
  lambda_function {
    lambda_function_arn = module.PersistGrantsGovXMLDB.lambda_function_arn
    events              = ["s3:ObjectCreated:*"]
    filter_suffix       = "/grants.gov/v2.OpportunityForecastDetail_1_0.xml"
  }
  1. Set the is_forecasted flag based on whether this is an opportunity or forecasted opportunity
jeffsmohan commented 1 week ago

@TylerHendrickson I'd love a quick review of the implementation plan before moving this to ready, thanks!

masimons commented 3 days ago

@TylerHendrickson @jeffsmohan Can I crash that^ review if it hasn't happened yet? :) editing your comment for fun lol