As a developer, I want to ingest data to on-demand SDS

[joshgarde]

In this ticket, we'll be building the ingest-to-sds worker described in the v2 architecture which takes granules from CMR and starts a job on the SDS via Orthello for ingestion; then monitors for completion.

AC: Successful data ingest from CMR to SWOT SDS

For this ticket, several sub-components will need to be built out:

• A worker lambda to run the cataloging process for the SDS
• A step function which periodically runs the worker lambda
• An authentication flow for authenticating to CMR from the lambda
• An authentication flow for authenticating to the SDS from the lambda
• An orthello API interface allowing for job submission
• An orthello API interface allowing for job status retrieval
• [OPTIONAL] An event-based callback interface for monitoring job status on the SDS side

[frankinspace]

After discussion, going to work with I&A to subscribe to CNM messages published from Cumulus. Needs:

• ingest-to-sds will be in its own repository
• SQS queue subscribed to I&A topic that publishes CNM on PIXC/PIXVec ingest
• Step function that launches new execution on new SQS message
  • lambda that translate CNM to sds api ingest job submission
  • wait loop that polls sds for job completion
  • on ingest job failure, log failure (and notify? email?)
  • on ingest job complete; end step function

@frankinspace Need to talk to I&A about allowing services to subscribe to topic @frankinspace Confirm with sds job throughput is supported

[frankinspace]

@joshgarde I spoke with @skorper about subscribing to the SNS topic for SWOT_L2_HR_PIXCVec_1.0 and SWOT_L2_HR_PIXC_1.0. She said it would be easier for them if we setup our own SNS topic and I&A will update some configs on their side so that the messages are published to our SNS topic (in addition to the others already there). They just need an ARN(s) of the topics we want to use.

[frankinspace]

Before I forget, we also need to ingest the XDF_ORBIT_REV files in addition to the PIXC/PIXCVec files. I'll try to also track down the details for that

[frankinspace]

These are the collection short names we need to ingest:

• SWOT_XDF_ORBIT_REV_FILE_1.0
• SWOT_L2_HR_PIXCVec_1.0
• SWOT_L2_HR_PIXC_1.0

[joshgarde]

So is the structure still generally the same where we have swodlr-api manage the SNS/SQS resources?

If so I'll open up a PR for that first in swodlr-api so I&A has the details while the cataloger is being worked on

[frankinspace]

@joshgarde yes, the only difference from our discussion is that swoldr will own and manage both the SNS and SQS resources, not just the SQS.

Sounds good, yeah the sooner we get the SNS topic defined the better I think

[joshgarde]

PR opened; awaiting review from @frankinspace

https://github.com/podaac/swodlr-ingest-to-sds/pull/2

[viviant100]

PR https://github.com/podaac/swodlr-ingest-to-sds/pull/2 has been approved and merged

[earthdata-github-robot]

(Comment from Tasktop Integration):

(Comment from Vivian Tang):

PR https://github.com/podaac/swodlr-ingest-to-sds/pull/2 has been approved and merged