Produce to SNS w/ . If it's from homepage then date is for that date's "top news"; otherwise in url there's already story and a date included.

[rivernews]

If we're fetching from home page, we actually don't need to fetch each page at this point, just the home page. SNS will cover "storing" the TODO fetch pages. But this requires SNS set longer retention day, and our development has to race against that - before message expires (14 days max, like SQS), we must finish implementing the part that subscribes and processes messages.

• [x] Pick a golang scraper (need to parse html) and hello world from the sfn lambda
  • [x] Shop here. We might use ferret, example quick code start here, but we need to understand context in Golang first.
    • Great beginner explanation.
  • [x] Install the scraper, try to run it!
  • [x] Golang runtime: send Slack message - can we reuse what we already have in head-k8s-service?
• [ ] Create SNS topic
  • [ ] Set permission Lambda->SNS
• [ ] Lambda logic: input (newsSiteIdentity)
  • [ ] Fetch homepage
  • [ ] S3 Archive html
  • [ ] Produce SNS
• [ ] SNS Attributes
  • Date
  • NewsSiteIdentity
  • newsTitle
  • Url
• [ ] Preliminary word cloud for landing page
  • [ ] Word cloud package to generate image? Shop.
  • [ ] How to send image to slack message?

Additional notes

• De-duplication: MD5 on the news content? (not the entire html, seems ads are always attached, and sidebar "related news" may change)

[rivernews]

A new proposal

Overall flow

1. API request(newsSiteLandingPage) --> parse landing page, archive on S3
2. Pull landing page from S3, --> extract all story links -->
3. Given links --> archive all stories on S3 --> extract all stories [rate control] --> store in DynamoDB w/ de-duplications
   • Can be concurrency=1; paginated, linked processing, expected to be a very long process and can't do it parallel.
4. Generate word cloud after waiting for all stories in landing page finished.
   • Pull from DynamoDB.query(newsSite=newsSite, date=date)
5. Optional - post to social media

Why splitting 1. and 2.?

• To lower cost from development time - we're losing data every day we haven't finished development.
• Otherwise if put 1 & 2 both in a sfn... imagine the I/O size will be huge flood with url links, some site around 90.
  • But even if you split, the paginated processing will still need the remaining urls to keep propagating to next self-linked processing. So why split 1 & 2 besides reason of development time?
• To deal with repetitively appearing news in landing pages? Not a valid reason, Combining together we can still do de-duplication; both case needs DynamoDB read/write.

Why splitting 2. and 3.?

• Then we can reuse 3., because each site structure is different (2.) - different ways to extract story links. But the fetching easy story part (3.) is the same
  • However, the way to extract each story may still be different. Only the S3 archiving part is same.

Splitting 2. and 3. is going to make 4. harder

• Since splitting eliminates the newsSite-story connection.
  • Unless we use a isLastStory signal on the last item, and use a strict FIFO queue that only contains that newsSite's link OR use a messageGroupId per newsSite. Then splitting 2. and 3. is okay.

Still, can some steps be combined and not split while fitting our goal?

[rivernews]

Focus: fetch and send to SQS for later processing

With that, we need both fetching the body AND parsing it. Ferret seems not easy to do this out of the box...its DOCUMENT method only accepts URL, not binary. Or maybe we can do a last try - see at its examples... nope, seems nothing related to archiving.

Let's shop another scraper that can match our needs...

• [x] goquery seems nice, for static website we can start from here.

Now we switched to GoQuery (which still doesn't have good doc, only one example, lack explanation on usage of most of the function). Next:

• [x] Output to Archive html to S3
  • [x] Install & code to upload to S3 bucket
  • [x] Permission to write to bucket
  • [x] Adapt io.ReadSeeker
  • [x] Call archive() in main.go.
  • [x] Send a html doc and test. Hard code a key.
• [x] Clean up zombie code

[rivernews]

S3 Archiving Improvement

• [x] Cronjob - do it daily or twice a day
  • Use event: schedule on lambda, then set up a lambda to send message to SQS, or just invoke step function
  • Can we set scheduler on step function directly?
• [x] Parallelism Experiment SQS FIFO polling lambda concurrency
  • If set 2 different message group, and send many messages... will AWS spin up two lambdas?
  • [x] Undo changes in consumer
  • [x] Add request in scraper lambda (golang) against api.shaungc.com.
  • [x] Add wait in scraper lambda (golang) to make it execute longer - so that the lambda lives longer and will help observe concurrency
  • [x] queue.tf increase the scraper lambda (golang one) concurrency
  • [x] Add 3 send messages, each with different messageGroupId
  • [x] Deploy
  • [x] Slack command, observe k8s nginx... got multiple different IPs or not?
• [x] Secretize news site URL. Should have some kind of configurability.
  • (Serverless.com) Managing secrets, API keys and more with Serverless
  • Deploying application configuration to serverless: Introducing the AWS AppConfig Lambda extension

Scraping Parallelism

Options

• SQS consumer, POC: fifo + messageGroupId
• Sfn + parallelism