Closed rivernews closed 1 year ago
But first, we had to re-cap dev env, last PR
story/main.go
is redundant, it's for a SQS story consumer (which I recall it never worked). Since we prefer using Sfn map we might not need it. But, we may leave it there for now? (Or...check it out to a independent branch...)stories/main.go
is intended for as a Sfn step. Input landing page S3 key, then it parse/scrape landing page, extract stories URLs, then send to SQS in chunks. It's partially redundant. We don't need the SQS sending in chunk part - replace this part by generating our metadata JSON.
stories_sfn.tf
is a new sfn. We can keep it as a single step (equivalent to a single lambda); of course we may use Sfn map but it's a separate process driven by other event so probably not going to share Sfn with that.stories_sfn
: stories/main.go
-> send sqs w/ (random) delay -> story consumer story/main.go
Then, we can get started on this task.
stories/main.go
.Do not merge into production yet, because some of the logic we may want to de-couple/couple.
main
: production branchdev
: dev branchCommit https://github.com/rivernews/media-literacy/pull/28/commits/0f7ab882a693a569e9ba456894f55ded7d8ca30c achieved this task.
Upstream: https://github.com/rivernews/media-literacy/issues/15#issuecomment-1221493722
POC
This is our very first time to leverage S3 trigger.
The EventBridge S3 notification could be useful:
Object Created
event type.Then, we need to filter by path. The same article gave example of filtering (event schema here):
s3://media-literacy-archives/{redacted}/daily-headlines/2022-08-21T12:15:42Z/landing.html
, we can filter by prefixs3://media-literacy-archives/{redacted}/daily-headlines
.TDD
How to test?
Action