[Spike] Harvesting pipeline in SSB

[nickumia-reisys]

Purpose

We want to test creating a brokerpak to provsion harvesting micro-services and have an external app facilitate triggers and connections, but we're not sure if it would work.

Given above question, conducting design + investigation + prototyping is needed to provide factual knowledge on future steps.

4 days of effort has been allocated and once compete, findings will be demonstrated and specific future actions will be decided.

Acceptance Criteria

• [x] GIVEN a queue pipeline has been created WHEN 4 days expires\ THEN the pipeline is shown via 0.5 hour presentation AND a recommendation is made on this approach for catalog harvesting

Background

Harvesting 2.0 proposed architecture

• Contending with https://github.com/GSA/data.gov/issues/4215

Sketch

Pre-work assumptions:

• [x] A new repository is created, specifically for this work (which may be thrown away if we decide later to abandon this approach).
• [x] Access to ssb-development account (AWS Credentials)

General Workflow:

• [x] Copy a skeleton of the brokerpak (either from an existing (solr) (eks) (smtp) or the template)
  • Full Disclosure: the template one may be too outdated for it to matter
• [x] Create an SQS Service (as brokerpak plan)
• [x] Create a Lambda Function (as brokerpak plan)
• Option 1:
  • [ ] Provision SQS
  • [ ] Provision Lambda function
  • [ ] Create SQS "binding"
  • [ ] Create Lamdba "binding"
  • [ ] Create a local script that connects the SQS binding to Lambda
  • [ ] Create a local script that puts things into SQS and checks outputs from the Lambda function
• Option 2:
  • [x] Provision SQS
  • [x] Create SQS "binding"
  • [x] Provision Lambda function (maybe with the SQS Binding as input?)
  • [x] Create Lambda "binding"
  • [x] Create local script that puts things into SQS and checks outputs from the Lambda function

(There are probably more options, but these two seem the most viable at the time of writing)

[nickumia-reisys]

Preliminary Notes/Observations:

• The Proof of Concept works but the design still leaves a lot of questions about implementation..
  • What does a "bind" for an SQS instance look like? (e.g. a restrictive policy on who can talk to it? a mapping/connection to another service? both? something else?)
  • What does a "bind" for a lambda function look like? (e.g. do we want to make an API gateway in front of the lambda function to interact with it and supply credentials there? do we want to have no binding and the function has everything it needs from the input to do its job?)
  • What are all of the inputs/outputs for each service?
  • SQS --> Lambda
  • Lambda --> SQS?
  • S3 --> SQS?
  • S3 --> Lambda?
  • Lambda --> S3
  • What other services do we need to implement?
• Overall, I think that this paradigm is powerful and robust, but there are quite a few options we'd need to design for and make available for proper customization when actually utilizing these services.

I will post an update shortly with the example test case as a reference.

[nickumia-reisys]

General overview of work and place to put additional notes.