mikwielgus / forum-dl

Scrape posts, threads from forums, news aggregators, mail archives, export to JSONL, mailbox, WARC
MIT License
68 stars 2 forks source link

Adding Time Sleep #18

Closed ismarapw closed 2 months ago

ismarapw commented 2 months ago

Time sleep is added to avoid the rate limit that sometimes appears in most scrapping processes.

Usage: forum-dl [url] -o [output.json] --time-sleep [second : int]

Example: forum-dl "https://forum.dronedeploy.com/c/feature-requests" -o test.json --time-sleep 3