MartinKBeck / TwitterScraper

Repository containing all files relevant to my basic and advanced tweet scraping articles.
197 stars 117 forks source link

Question about scraping last day of the month #11

Closed JacopoGiannetti closed 2 years ago

JacopoGiannetti commented 2 years ago

Hi! First of all, thank you very much for this tool, this is helping me with my dissertation so much! I'm having a problem while scraping tweets about a certain topic for a specified period of time. The problem is that when I try to get tweets like from 01/05/2019 to 31/05/2019 I only get tweets up to the 30th. For my dissertation I needed 10k tweets a day for the past 3 years, I built a function and it worked perfectly, I extracted around 300k tweets a month but in the end, I found out that I always miss the last day of each month. In order to add all the missing days, I need to use the tool to download tweets from the last day of each month, but from what I understood I always specify a "since" date and an "until" date. So for getting 1st May tweets I need to specify from 01/05/19 to 02/05/19 and I will get tweets from the 1st of May. The problem is that if I am dealing with the 31st of May I cannot specify an until date as it would be the 32nd that doesn't exist.. Am I missing something? How can I just get tweets from a specific day? ps. if I set the same date as since and until date it doesn't work Thank you in advance