Open lapp0 opened 4 years ago
@christiangfv please try what I just pushed
I can confirm that this method works quite well. I am wondering if using helium won't solve the geckodriver installation issue? As helium ships with common webdrivers.
See helium https://github.com/mherrmann/selenium-python-helium
I tried this suggestion from a friend and was able to make the geckodriver work (in Ubuntu 18.04 with Firefox 80.0.1):
wget https://github.com/mozilla/geckodriver/releases/download/v0.27.0/geckodriver-v0.27.0-linux64.tar.gz
tar -xvf geckodriver-v0.27.0-linux64.tar.gz
sudo mv geckodriver /usr/bin
@lapp0 does not change, I have the same 502 error. I try it with proxy, no proxy, in firefox and chrome driver and it's the same
@christiangfv thats unfortunate. Perhaps someone more talented than me could update the docker image to install gnu/linux, selenium, geckodriver, and firefox. I'm not sure what your specific issue is. Perhaps you could try running it in a debian virtual machine?
(also, I assume you pulled and ran python3 setup.py install
?)
Yes @lapp0 , I have done it, now I will test in a virtual machine with Ubuntu.
@lapp0 Works in linux Ubuntu!! Ubuntu is always a faithful ally. But I have a new problem, you know some this?
mi code:
tweets = twitterscraper.get_query_data('realDonaldTrump',poolsize=1,begindate=dt.date(2020, 9, 20),limit=10)
print(tweets)
and this function return a void list. but when I debug, I can check that take a correctly tweet.
Thanks for testing @christiangfv !
I made a mistake in the date filtering function I pushed today so it doesn't allow single-day ranges. Pushing a fix.
With this you can solve
but it is already resolved with your code jajjaja
Thanks @lapp0, Hey friend, do you know if this implementation works on a server with ubuntuServer or other?
@christiangfv Good to hear!
I don't see any reason why this wouldn't work on ubuntu server
I got this problem and i don't know why
I use "From within Python"
I got this problem and i don't know why I use "From within Python"
I had run this, but it seemed no effect
@Buaasinong Are you in the correct directory? What is your operating system? in linux you need the sudo command.
Are you in the correct directory? What is your operating system? in linux you need the sudo command.
Oh, i'm using windows.And i think my directory is correct.
Are you in the correct directory? What is your operating system? in linux you need the sudo command.
Oh, i'm using windows.And i think my directory is correct.
I try to run setup.py directly ,and get this
@Buaasinong The correct command for the installation is Python3 setup.py install
. You need add argument install
@Buaasinong The correct command for the installation is
Python3 setup.py install
. You need add argumentinstall
Yes! I used python setup.py install(because python3 setup.py install is still no result).But it still don't work. Could you give me a picture about successfully running in terminal?
@Buaasinong The correct command for the installation is
Python3 setup.py install
. You need add argumentinstall
Yes! I used python setup.py install(because python3 setup.py install is still no result).But it still don't work. Could you give me a picture about successfully running in terminal?
OH,I think
python setup.py install
is not useful.sad
@Buaasinong are you running python2? What is python --version
?
Would be really helpful if someone got the docker working with selenium so we could avoid these version and platform specific issues.
Hi @lapp0, I have noticed that when I retrieve the tweets, they do not appear in order. I only need the last 20 tweets from an account in order. For this I implement changes in
and the new link that i used is:
INIT_URL = 'https://twitter.com/search?q={q}&src=typed_query&f=live'
and finally, for sorted tweets, I sorted previusly the dict with tweets in:
@Buaasinong are you running python2? What is
python --version
?Would be really helpful if someone got the docker working with selenium so we could avoid these version and platform specific issues.
python 3.7 ,but it‘s 32bits
Hi Bro ,I got a new problem like you in browser @christiangfv ,and I want to disable proxy for using --disableproxy,there is an error in my terminal
Hi @Buaasinong, the problem is that the --disableproxy or -dp argument is not implemented. I had the problem on macOS 10.15.6, I think it could be system restrictions. I solved it by installing on Ubuntu linux
--disableproxy
hasn't been merged into this branch yet, but is in master. However, I don't think that's the problem. I think it's a problem specific to selenium that don't involve the proxies at all.
I have pushed untested docker changes to install geckodriver, however they're pretty straightforward. @Buaasinong could you try to use the Dockerfile
to build and run?
Bug find:
The following query has variable results: https://twitter.com/search?q=foo+bar+since%3A2017-11-11+until%3A2017-11-13
Sometimes it includes foobar is being so I'm out the first round
and sometimes it doesn't.
I can't figure out what conditions result in this being the case, sometimes there are 20 results, sometimes there are 40, sometimes 57. Perhaps the only option is to require multiple passes, unfortunately.
You can try this manually by going to that url and retrying multiple times until foobar is being so I'm out the first round
is missing (or present) to see its variable presence.
This is tough to test.
--disableproxy
hasn't been merged into this branch yet, but is in master. However, I don't think that's the problem. I think it's a problem specific to selenium that don't involve the proxies at all.I have pushed untested docker changes to install geckodriver, however they're pretty straightforward. @Buaasinong could you try to use the
Dockerfile
to build and run?
Thanks Bro, I will try
Hi @Buaasinong, the problem is that the --disableproxy or -dp argument is not implemented. I had the problem on macOS 10.15.6, I think it could be system restrictions. I solved it by installing on Ubuntu linux
Wow, It's a bad news.But still appreciate.
Upon running a query, I get the following error: "ModuleNotFoundError: No module named 'twitterscraper.ts_logger"
@Bakdaulet1 I merged master and that broke some stuff. Most recent push should resolve.
I ran this 100 times:
query_js.get_query_data(begindate=dt.date(2019, 11, 11), enddate=dt.date(2019, 11, 13), poolsize=3, lang='en', query='foo bar')
I then counted the number of tweets recieved each time. Here is the mapping of "number of tweets" -> "number of times this number of tweets were returned"
41: 9
55: 16
70: 20
73: 1
74: 1
83: 3
84: 50
So exactly half the time we got 84 tweets.
I'm running this experiment again with multiple passes for each day (unfortunate but probably workable solution), will update this comment with results.
https://github.com/lapp0/twitterscraper/blob/faf75afb7eff15a04df471a0b651011b67db85fa/Dockerfile#L4 so the firefox install here in the docker file doesnt work as is , https://hub.docker.com/r/selenium/node-firefox/dockerfile should perhaps maybe use the officially maintained selenium//firefox dockerfile as a base ??
@webcoderz you probably know docker better than me, as I use a different packaging/containerizing solution (nix). Would you be willing to make the change?
would you recommend doing 2 docker services communicating over a docker network vs one container vs single container?
@webcoderz IMHO it'd be better to have a single docker container.
I am getting the following error:
INFO:twitterscraper:Scraping tweets from https://twitter.com/Bakdaul52729738 INFO:twitterscraper:Using proxy 191.96.42.80:8080 INFO:twitterscraper:Got 0 tweets from username Bakdaul52729738
Number of passes -> % of time it collects every single tweet
get_query_data(begindate=dt.date(2019, 11, 11), enddate=dt.date(2019, 11, 13), poolsize=3, lang='en', query='foo bar')
(expect 84 total)
get_query_data(begindate=dt.date(2018, 5, 5), enddate=dt.date(2018, 5, 7), poolsize=3, lang='en', query='trump')
(expect ?? total)
(running, will update)
@Bakdaulet1 please share your command/code
@Bakdaulet1 please share your command/code
twitterscraper Bakdaul52729738--user -o tweets_username.json
@Bakdaulet1 the legacy endpoint is broken. To use the changes in this merge request you need the --javascript
(or -j
) argument. Also please merge the following changes I'm about to push before running.
would also be nice to have a function that scrapes by twitter id....
you can use this twitter_id="1308322643054034944" "https://twitter.com/anyuser/status/{}".format(twitter_id)
https://twitter.com/anyuser/status/1308322643054034944 will take you to the correct tweet , alls you need is the twitter id
@webcoderz good idea! Although I think that's out of scope for this merge request. Feel free to make your own.
For a programatic example, see test_simple.py
https://github.com/taspinar/twitterscraper/pull/302/files#diff-9ccfdcc9b53fb500cf131ea181f733e8
ok had found a error in my Dockerfile will try again and let you know, just pushed the change and PR (had symlink to just geckdriver but not to selenium wire
@lapp0 submitted PR that dockerfile tested works, puts executable in /bin for geckodriver, was wondering can also add the other browser drivers as well if u wanted to do rotating browsers for improved proxying capability or something like that
IMO, adding new browsers will just increase complexity with little to no benefit. I don't think twitter is going to throttle/ban Firefox.
The biggest problem right now is memory and CPU, so if chrome is more efficient browsing twitter, we might want to go that way. Although, I think installing ublock origin in firefox would be a better decision for that.
Ran into this for a big query:
Going to experiment with a throttling mechanism avoid
i think can do that all via docker too will look into it..
That'd be great, we might even be able to set custom ublock rules to block irrelevant twitter endpoints and speed up scraping.
@lapp0 I noticed this in the latest commit... And I am curious, with the return
keywords in the if...else
conditional statement, will the code ever go further to retry
?
@smuotoe Yeah, that entire function needs to be cleaned up. It's slowly built technical debt as I've handled more and more edge cases.
Good catch, pushing a fix.
@webcoderz thats too bad. Do you know if chromium has these capabilities?
A variety of issues have recently arisen due to Twitter disabling their "Legacy" API, breaking twitterscraper:
To fix this, I re-implemented
query.py
using Selenium, allowing twitterscraper to programatically control a background (headless) Firefox instance.Additionally, I refactored
query.py
(nowquery_js.py
) to be a bit cleaner.Based on my testing, this branch can successfully download tweets from user pages, and via query strings.
How to run
Please test this change so I can fix any bugs! 1) clone the repo, pull this branch 2) install selenium dependencies (geckodriver and firefox) https://selenium-python.readthedocs.io/installation.html 3) enter twitterscraper directory,
python3 setup.py install
4) run your queryIf you have any bugs, please paste your command and full output in this thread!
Improvements
get_query_data
(all tweets / metadata from a specific query) andget_user_data
(all tweets / metadata on a users page).--user
wouldn't get all of a users tweets and retweets due to a limitation in twitters scrollback for a given user. Now a workaround enables retrieving of tweets and retweets for a specific user via a custom search:f'filter:nativeretweets from:{from_user}'
query_user_info
brokenNotes
pos
was removed - now the browser is used to storepos
state implicitly--javascript
and-j
now decide whether to usequery.py
orquery_js.py
Problems
limit
no longer works, though this should be relatively easy to fix if sufficiently desired~ (limit has now been implementedquery_user_info
andquery_user_page
haven't been converted to use selenium, they don't work right now. However, this data is returned as part of the metadata mentioned in Improvements bullet 2pip install
. However use of docker can alleviate this.