shaikhsajid1111 / twitter-scraper-selenium

Python's package to scrap Twitter's front-end easily
https://pypi.org/project/twitter-scraper-selenium
MIT License
299 stars 46 forks source link

How to show progress for the scraping process? Right now there is no indication that it works until it is all done. #61

Open richlysakowski opened 1 year ago

richlysakowski commented 1 year ago

I am still learning to use the package. It is VERY nice in terms of functionality.

The delays before seeing anything can be 2-3 minutes or longer, depending on how many tweets are being fetched.

Is there an easy way to monitor progress without slowing down the scraping process?

I see that logging is built in. Where are the log files stored?
Can the log be streamed to another terminal in my IDE? I am using PyCharm Pro.