(1/3) Create S3 landing page file trigger - invoke lambda - generate JSON

[rivernews]

Upstream: https://github.com/rivernews/media-literacy/issues/15#issuecomment-1221493722

POC

This is our very first time to leverage S3 trigger.

The EventBridge S3 notification could be useful: Object Created event type.

Then, we need to filter by path. The same article gave example of filtering (event schema here):

• Object size
• Key prefix matching Our landing page objects stores at s3://media-literacy-archives/{redacted}/daily-headlines/2022-08-21T12:15:42Z/landing.html, we can filter by prefix s3://media-literacy-archives/{redacted}/daily-headlines.
• Terraform module to try out.

TDD

How to test?

1. Manually upload a landing page html to S3 to trigger
2. Observe the triggered lambda is executed.

Action

1. [ ] Setup empty pipeline just to make sure the platform works.

[rivernews]

But first, we had to re-cap dev env, last PR

• [x] Check out all the local changes, make sure push all changes to GitHub.
• [x] Any redundant code we want to clean up in the PR?
  • story/main.go is redundant, it's for a SQS story consumer (which I recall it never worked). Since we prefer using Sfn map we might not need it. But, we may leave it there for now? (Or...check it out to a independent branch...)
  • stories/main.go is intended for as a Sfn step. Input landing page S3 key, then it parse/scrape landing page, extract stories URLs, then send to SQS in chunks. It's partially redundant. We don't need the SQS sending in chunk part - replace this part by generating our metadata JSON.
    • stories_sfn.tf is a new sfn. We can keep it as a single step (equivalent to a single lambda); of course we may use Sfn map but it's a separate process driven by other event so probably not going to share Sfn with that.
    • Previously this PR is trying to do: slack command -> stories_sfn: stories/main.go -> send sqs w/ (random) delay -> story consumer story/main.go
• [x] Can we extract irrelevant code out of the PR as a new branch? So we can preserve all the POC still.
• [x] Set up Golang dev env: auto-complete in VSCode.

Then, we can get started on this task.

Directly work on this prev PR for our metadata JSON

• [x] We can leave the slack command as-is. Perhaps also keeping the Sfn not sure; the change: event bridge trigger Sfn or lambda
  • [x] Having Terraform not providing Sfn interface so we'll just trigger stories/main.go.
  • [x] Apply in dev to try it out
  • [x] TDD: Test by uploading a landing page html to S3, and observe receiving a slack message. The msg should contain the S3 event object.
• [x] Check In: we can trigger upon S3 object creation and successfully run a skeleton compute env (Sfn or lambda)
• https://github.com/rivernews/media-literacy/pull/27 reached this milestone above.
• [x] Generate metadata JSON right after extracting stories from landing page. PR: https://github.com/rivernews/media-literacy/pull/26 ... commit: https://github.com/rivernews/media-literacy/pull/28/commits/0f7ab882a693a569e9ba456894f55ded7d8ca30c

[rivernews]

Do not merge into production yet, because some of the logic we may want to de-couple/couple.

• main: production branch
• dev: dev branch

[rivernews]

Commit https://github.com/rivernews/media-literacy/pull/28/commits/0f7ab882a693a569e9ba456894f55ded7d8ca30c achieved this task.