Closed Purefreeman closed 6 years ago
Yes, as stated in the README, media-scraper
cannot download the BLOB URL videos in Twitter currently. I'll keep working on it but no guarantee when to complete.
Ok, sorry for the misunderstanding
Hi @Purefreeman,
I've updated the mediascraper.twitter
and now it supports downloading both photos and videos from Twitter.
Tried multiple times with the new update, but i keep getting this error
File "C:\Users\dolap_000\AppData\Local\Programs\Python\Python36-32\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\dolap_000\Desktop\media-scraper-master\mediascraper\twitter.py", line 18, in
Could you please provide an example Twitter username so that I can reproduce the problem and fix it? I've tested media-scraper
on the Twitter account realDonaldTrump
which contains a bunch of photos and videos.
python -m mediascraper.twitter purefreeman Starting PhantomJS web driver... Web driver ".\webdriver/phantomjsdriver_2.1.1_win32/phantomjs.exe" not found. Start downloading the web driver... Web driver ".\webdriver/phantomjsdriver_2.1.1_win32/phantomjs.exe" has been downloaded successfully. .\webdriver/phantomjsdriver_2.1.1_win32/phantomjs.exe C:\Users\dolap_000\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\phantomjs\webdriver.py:49: UserWarning: Selenium support for PhantomJS has been deprecated, please use headless versions of Chrome or Firefox instead warnings.warn('Selenium support for PhantomJS has been deprecated, please use headless ' Logging in as "Kheshig"... Crawling... Traceback (most recent call last): File "C:\Users\dolap_000\AppData\Local\Programs\Python\Python36-32\lib\runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "C:\Users\dolap_000\AppData\Local\Programs\Python\Python36-32\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\dolap_000\Desktop\media-scraper-master\b\media-scraper-master\mediascraper\twitter.py", line 18, in
tasks = scraper.scrape(username) File "C:\Users\dolap_000\Desktop\media-scraper-master\b\media-scraper-master\mediascrapers.py", line 398, in scrape img_url, vid_url = get_twitter_video_url(li['data-item-id']) File "C:\Users\dolap_000\Desktop\media-scraper-master\b\media-scraper-master\util\twitter.py", line 13, in get_twitter_video_url return post['posterImage'], post['track']['playbackUrl'].rsplit('?', 1)[0] KeyError: 'posterImage'
I've updated it to handle the json parsing exception. Please take a look.
Yes it seems to be working. But it doesn't download some gifs
Could you give me an example? In my observation, the gifs are rendered as mp4s in Twitter posts and they are crawled and downloaded by mediascraper
.
I can't seem to find a good account to show this or recreate it. Thanks
As stated in the title, i can only download pictures. i download 4 different accounts and it still has the same result