Open yarnball opened 7 years ago
How many ids are you looking to scrape? If it's just a few hundred then I'd use the exit code of the instagram-screen-scrape comments
command to retry, up to a maximum number of times.
If it's millions, then I'd put together a set of workers in JS & use RabbitMQ for task distribution / retries. The command line isn't that efficient for scraping (it requires starting up a new node process for each scrape and can't reuse the http connection between them). The CLI is just there because it's quick to setup for little tasks.
Hi,
Great work on this. So I've got a .txt file with a list of mediaID's I'd like to scrape the comments from.
However, I can only do them one-at-a time in your script.
I don't know CoffeeScript. How is this possible in your repo?
I tried with a BashScript- however there are often 404 errors on the comment scrapes. It often works if I "re-attempt" the scrape. Is there a "proper" way to do this?
Here's a copy of my Bash file