Closed rivernews closed 1 year ago
Primary table: just UUID
Landing page table:
isMetadataEverComputed
is the important one.landing
lambda manually in AWS portal, so it can fetch one single landing HTML, and create DDB item.
landing_metadata_cronjob
lambda manually. Observe DDB query successfully getting landing page item, and output the metadata.
isDocTypeWaitingForMetadata
removed, event added
list_append
only works with two lists, not one list one item.stories
story
in parallel, and finalizer lambda stories_finalizer
executed
stories-...
prefix there that shouldn't be there, check the dev bucket.Basically: turn S3 object(s) into a brand new DDB item.
landing
s3://media-literacy-dev-archives/{redacted}/daily-headlines/2021-08-21T00:11:53Z/...
, we can probably scan by time range.To kick start,
Avoid writing unnecessary code. This one-time thing is going to be used really rare after this first trigger. Leverage the S3 trigger + "move/copy" feature in S3 bucket. The flow could be like:
PutItem
from landing
to another lambda. S3 trigger landing.html
invoke this lambda.
s3:ObjectCreated:Put
.s3:ObjectCreated:Copy
metadata.json
as well. Now: just copy the landing.html
one by one, don't copy by the entire directory!
🛑 ERROR: operation error DynamoDB: Query, https response error StatusCode: 400,
RequestID: 653RRF4ESDBN7P72AER3M9DAURVV4KQNSO5AEMVJF66Q9ASUAAJG, api error
ValidationException: One or more parameter values are not valid.
A value specified for a secondary index key is not supported.
The AttributeValue for a key attribute cannot contain an empty string value.
IndexName: s3KeyIndex, IndexKey: s3Key | 2022/09/30 08:57:47
There are quite big cost implication, however we don't know exact the amount of $$ we need to pay yet. But moving forward it's time to think about the fast track issues and cost saving issues. We should have another issue address these, since they are out of scope and no longer about achieving one-time batch processing.
For now, we will disable cronjob and pause the pipeline. Next time, we may copy over the stories to prod for reuse. Once we have the fast track feature https://github.com/rivernews/media-literacy/issues/41, those will be skipped and we won't lose the computation outcome of these days.
Better way to run them all
landing.html
trigger - just write into DDB https://github.com/rivernews/media-literacy/pull/34. Move the metadata computing part into new cronjob below.s3Event
to pulling from DDB. (do a query; canlimit=1
for our "slow start" purpose) https://github.com/rivernews/media-literacy/pull/35isMetadataEverComputed
pipelineEvents
.Reference
Proper Throttling
It'll be best to reuse the Sfn, but limit the amount of concurrent sfn execution; overall we should aim at 5~100 concurrent lambdas but nothing more. Ideal if we can throttle <1 request / 2s.
But to truly keep a low profile it's best to span the time across hours, if not days.
Moving forward
Daily cronjob should automatically trigger our new S3-driven pipeline. Any other concerns?
[x] Staging drill: parameter all switch to prod; then copy all prod landing page over to dev S3 bucket to test.
200
, why? Because the landing page didn't have enough stories! Look at the db, it's88
only so.Improvements - only do low hanging fruit at this point! Don't do complicate task
lastEventName
to query (but then it could be similar to scan)? Or just a opposite toisDocTypeWaitingForMetadata
, likeisDocTypeMetadataDone
.[ ] Do the same above for prod (ready!)