Explore news site

[rivernews]

• [x] What site to scrape, how deep?
  • [x] Then generate a word cloud image
  • [x] Store image on S3, generate presign url
  • [x] Post image to slack
• [x] Use a queue to scrape site, so that the site doesn't get bombard.
• [x] Word cloud - when to trigger? Need to wait all relevant page done their work.
• [ ] AWS Batch or Glue for processing? Even if we can use lambda
  • [x] Golang can shine here, it's really good at parallel processing

It's important to make above modularize, so it's better for testing, reusing, parallel execution, etc.

Module: scraper (Golang)

• [ ] Input: URL
• [ ] Queue time spreader - to avoid bombard website
• [ ] Side effect: depth=1 recursive fetch, store raw file on S3 ... parallel processing
• [ ] Flow control: wait till all scraper-session in queue done their job, then we can proceed to module parsed/formatted.
  • [ ] 🔥 what mechanism to notify proceeding? SQS vs Step function.

Module: parser & formatter

• [ ] Input: S3 raw files
• [ ] Operation: parse formatted story doc.
• [ ] Side effect: can use DynamoDB since processing word cloud need performance; and we can clear it since we have raw file backup.

Module: word cloud reducer

• [ ] Input: load text of all grouped pages from DynamoDB
• [ ] Produce word cloud image
• [ ] Store image on S3, generate presign url
• [ ] Publish on slack

Module: time series scraper

• [ ] Meter anchor - duplication prevention
• [ ] Auto follow

Module: publisher

• [ ] FB SDK

---

Scraping - Golang runtime environment

• Hierarchy for word cloud

  • News site > date - today's top news > stories

• Hierarchy for news stream

  • News site > stories

• SNS ideas: fan in (url), fan out 1) for generating word cloud 2) for porting to fb

  • Perfect use case for SNS's multiple fan out

---

Let's split into two parts - 1) fetch html 2) rest of the processing

• [x] #13
• The rest of the work as below can be done later when subscribing to SNS
  • [ ] (Urgent) DynamoDB L1 caching (priority - serve as a backup for SNS message retention limit)
    • Attributes: date time, url, newsSiteIdentifity, newsTitle, s3Path, lastPipelineStep(queued, fetchedToS3, fbPorting, wordCloud, done)
  • [ ] (Urgent) S3 copy of the html page
  • Processing: word cloud - can wait, just pull from DynamoDB.
  • Fb porting

[rivernews]

POC: Parallel Batch Scraper with Golang

• [x] Let's use a golang lambda in step func.
  • [x] First install golang, before that install a version manager. GVM, but last maintenance is 2019, while having 6K stars. g seems neat, .3K stars, yet actively maintained. There're also gobrew, goup, around .1K stars.
  • [x] Use the built-in go mod to manage dependency, also best practice of naming the module.
  • [x] Allow sfc to call lambda
• [x] SQS - While golang is powerful to spin up thousands of goroutine, we need to rate limit the request to avoid bombarding the website. SQS long poll max interval is 20s - way enough.
  • [x] Create SQS: SQS from serverless.tf
    • [x] Set interval, etc
    • [x] Set batch size
  • [x] Connect SQS: APIGateway -> SQS -> Step func
  • [x] Consumer lambda: enable long polling by setting event source mapping: start step func
  • [x] API Gateway send message
  • [x] Consumer lambda: concurrency control Turns out that we don't need this... at least the scraping part.
  • [x] Clean up original sfn starting
• [x] SQS long polling troubleshooting - looks like did not wait, when sending 3 requests in a roll, consumer invoked 3 times almost at the same time, doesn't look at interval = 3 for long polling.
  • Even with concurrency limiting, long polling wait time, still consumer lambda invoked too fast, 10 within 4 sec.
  • But at least sleep in your consumer lambda can slow down the flow right? Given concurrency=1.
  • [x] Message not deleted after consumer lambda finishes -- because lambda default timeout is 3s! Increased to longer now

[rivernews]

Still work remaining