Closed rivernews closed 3 years ago
Overall flow
Why splitting 1. and 2.?
Why splitting 2. and 3.?
Splitting 2. and 3. is going to make 4. harder
isLastStory
signal on the last item, and use a strict FIFO queue that only contains that newsSite's link OR use a messageGroupId
per newsSite. Then splitting 2. and 3. is okay.Still, can some steps be combined and not split while fitting our goal?
With that, we need both fetching the body AND parsing it. Ferret seems not easy to do this out of the box...its DOCUMENT
method only accepts URL, not binary. Or maybe we can do a last try - see at its examples... nope, seems nothing related to archiving.
Let's shop another scraper that can match our needs...
Now we switched to GoQuery (which still doesn't have good doc, only one example, lack explanation on usage of most of the function). Next:
io.ReadSeeker
archive()
in main.go
.api.shaungc.com
.queue.tf
increase the scraper lambda (golang one) concurrencyOptions
If we're fetching from home page, we actually don't need to fetch each page at this point, just the home page. SNS will cover "storing" the TODO fetch pages. But this requires SNS set longer retention day, and our development has to race against that - before message expires (14 days max, like SQS), we must finish implementing the part that subscribes and processes messages.
ferret
, example quick code start here, but we need to understandcontext
in Golang first.Additional notes