neon-ninja
commented
3 years ago Give latest master a try. I added a few things:
- I added an argument for limiting to the last n days, like so:
-d DAYS_LIMIT, --days-limit DAYS_LIMIT Number of days to download - I added an argument for storing the last pagination URL to a file, and to resume from that file:
-rf RESUME_FILE, --resume-file RESUME_FILE Filename to store the last pagination URL in, for resuming - I added an argument for specifying a comma separated list of keys:
-k KEYS, --keys KEYS Comma separated list of which keys or columns to return. This lets you filter to just your desired outputs. - You can redirect stderr to a file to achieve this. In bash, you can say
2>error.log - I added a new column called
page_id - I added a new argument:
-m MATCHING, --matching MATCHING Filter to just posts matching string
Putting it all together:
facebook-scraper Nintendo -vvv -f - -fmt json -d 3 -ner -rf resume -k post_id,time,page_id,likes -m Metroid 2>error.logoutputs:
[
{
"post_id": "4549796628438088",
"time": "2021-09-30 09:00:02",
"likes": 341,
"page_id": "119240841493711"
},
]Also, results now stream to the file as the scraping is running, instead of collecting everything and then writing everything at once.
wgicio
I have a few ideas which should make peoples lives easier when scraping
1) It would be great to be able to have a CLI option to stop scraping once the posts reach a certain date. Would be useful if someone wants to scrape the last weeks worth of posts
2) The ability to start the scraping from a specific starting point, either a date, or from when a previous scrape stopped due to a temporary block by Facebook
3) A way to specify what needs to be scraped eg --HQimages 1 --LQimages 0 --videos 0 -- reactions 0 -- factcheck 0
4) save the debug log alongside the filename for each scrape facebook-scraper --filename nintendo_page_posts.csv --verbose would also save a file a file called nintendo_page_posts.log
5) an extra column in the saved data with the group / page ID would also be nice for identifying the source when combining multiple scrapes
6) only save the record if a keyword is present in the post text
thanks!