elvisyjlin / media-scraper

Scrapes all photos and videos in a web page / Instagram / Twitter / Tumblr / Reddit / pixiv / TikTok
MIT License
371 stars 49 forks source link

Enhancement: deactivated accounts #5

Open Purefreeman opened 6 years ago

Purefreeman commented 6 years ago

I have been using a bat file to run the program; which contains multiple twitter account. The problem is that if an account was to deactivate their account it would broke the program, stopping it. This shouldn't be the case as the program should be able to skip it and continues to crawl the other accounts; and at the end it should say the accounts that longer exist.

error produce when you run a bat file that contains an account that has deactivated

Starting PhantomJS web driver...
.\webdriver/phantomjsdriver_2.1.1_win32/phantomjs.exe
C:\Users\dolap_000\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\phantomjs\webdriver.py:49: UserWarning: Selenium support for PhantomJS has been deprecated, please use headless versions of Chrome or Firefox instead
  warnings.warn('Selenium support for PhantomJS has been deprecated, please use headless '
Logging in as "Kheshig"...
Crawling...
Traceback (most recent call last):
  File "C:\Users\dolap_000\AppData\Local\Programs\Python\Python36-32\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\dolap_000\AppData\Local\Programs\Python\Python36-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\dolap_000\Desktop\media-scraper-master\mediascraper\twitter.py", line 18, in <module>
    tasks = scraper.scrape(username)
  File "C:\Users\dolap_000\Desktop\media-scraper-master\mediascrapers.py", line 384, in scrape
    done = self.scrollToBottom()
  File "C:\Users\dolap_000\Desktop\media-scraper-master\mediascrapers.py", line 87, in scrollToBottom
    last_height, new_height = self._driver.execute_script("return document.body.scrollHeight"), 0
  File "C:\Users\dolap_000\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 627, in execute_script
    'args': converted_args})['value']
  File "C:\Users\dolap_000\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 312, in execute
    self.error_handler.check_response(response)
  File "C:\Users\dolap_000\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 237, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: {"errorMessage":"Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: \"script-src https://abs.twimg.com https://ssl.google-analytics.com https://ajax.googleapis.com http://www.google-analytics.com about:\".\n","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"112","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:50385","User-Agent":"Python http auth"},"httpVersion":"1.1","method":"POST","post":"{\"script\": \"return document.body.scrollHeight\", \"args\": [], \"sessionId\": \"afc32870-757b-11e8-9639-3b1da092f52a\"}","url":"/execute","urlParsed":{"anchor":"","query":"","file":"execute","directory":"/","path":"/execute","relative":"/execute","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/execute","queryKey":{},"chunks":["execute"]},"urlOriginal":"/session/afc32870-757b-11e8-9639-3b1da092f52a/execute"}}
Screenshot: available via screen.