Open AleksandrovichK opened 3 years ago
Hi, sorry for the late response. First of all, set interval
with a value smaller than or equal to the period of time you want to scrap (1 in your case). Seconde, what does the code print while scraping ? Is there somethin like "Tweet... Found .." ?
For the case of user_info, you need to set your credentials first.
Okay so it appears to be I started to figure out what's wrong
First of all, I have switched off headless mode (so I can see the browser).
If I specify promixity=True
then Twitter doesn't show anything in the search bar. So there is something wrong with the request in the address bar.
If I choose to run WITH headless
mode, then I have:
Scraping on headless mode.
looking for tweets between 2021-04-29 and 2021-04-30 ...
path : https://twitter.com/search?q=(from%3Aelonmusk)%20until%3A2021-04-30%20since%3A2021-04-29%20%20-filter%3Areplies&src=typed_query
scroll 1
scroll 2
looking for tweets between 2021-04-30 and 2021-05-01 ...
path : https://twitter.com/search?q=(from%3Aelonmusk)%20until%3A2021-05-01%20since%3A2021-04-30%20%20-filter%3Areplies&src=typed_query
scroll 1
scroll 2
looking for tweets between 2021-05-01 and 2021-05-02 ...
path : https://twitter.com/search?q=(from%3Aelonmusk)%20until%3A2021-05-02%20since%3A2021-05-01%20%20-filter%3Areplies&src=typed_query
scroll 1
scroll 2
But when I choose WITHOUT headless
mode (which is obviously less preferable in production-like activities) then I have some tweets found.
looking for tweets between 2021-04-29 and 2021-04-30 ...
path : https://twitter.com/search?q=(from%3Aelonmusk)%20until%3A2021-04-30%20since%3A2021-04-29%20%20-filter%3Areplies&src=typed_query
scroll 1
scroll 2
scroll 3
looking for tweets between 2021-04-30 and 2021-05-01 ...
path : https://twitter.com/search?q=(from%3Aelonmusk)%20until%3A2021-05-01%20since%3A2021-04-30%20%20-filter%3Areplies&src=typed_query
scroll 1
scroll 2
looking for tweets between 2021-05-01 and 2021-05-02 ...
path : https://twitter.com/search?q=(from%3Aelonmusk)%20until%3A2021-05-02%20since%3A2021-05-01%20%20-filter%3Areplies&src=typed_query
Tweet made at: 2021-05-01T21:36:22.000Z is found. <---- FOUND ONE
Tweet made at: 2021-05-01T21:49:50.000Z is found. <---- FOUND ONE
scroll 1
scroll 2
scroll 3
looking for tweets between 2021-05-02 and 2021-05-03 ...
path : https://twitter.com/search?q=(from%3Aelonmusk)%20until%3A2021-05-03%20since%3A2021-05-02%20%20-filter%3Areplies&src=typed_query
Tweet made at: 2021-05-02T03:13:36.000Z is found. <---- FOUND ONE
scroll 1
scroll 2
scroll 3
So it seems like an issue. The ability to use it in headless mode - is a very important thing.
I've made an experiment. I ran the scrapping with exactly the same parameters 3 times and saved shapes of the results. Here they are:
(4, 11)
<- the first run, I kept the browser window active all the time
(2, 11)
<- opened it a couple of times and then covered the window with another
(0, 11)
<- window with the scrapper was hidden
It seems like Twitter reacts to users' actions.
I'm sorry, any updates? @Altimis
I have the same issue, it works only with headless=False
Hey guys ! Facing the same issue, after using headless = False, getting blank dataframe.
data = scrap(start_date="2021-05-01", max_date="2021-05-02", from_account = 'elonmusk', interval=1, headless=False, display_type="Top", save_images=False, filter_replies=True, proximity=True)
Getting output as - looking for tweets between 2021-05-01 and 2021-05-02 ... path : https://twitter.com/search?q=(from%3Aelonmusk)%20until%3A2021-05-02%20since%3A2021-05-01%20%20-filter%3Areplies&src=typed_query&lf=on scroll 1 scroll 2 scroll 3
Whats wrong here?
Facing the same issue. Works with Headless=False
otherwise when set to True
I just see that it is scrolling and not picking up any of the tweets.
Hey guys!
I tried to reproduce just the same code as you use in examples:
After running the code, the variable
users_info
containsNone
.Yet I tried this:
And the variable data contains empty dataframe.
Maybe I'm doing something fundamentally wrong?
My platform: MacOS Big Sur A version of Scweet: 1.0