search

cwoac / thingy_grabber

Script for archiving thingiverse things
MIT License
39 stars 11 forks source link

Timeout loading page after 300000ms error #12

Closed joebywan closed 4 years ago

joebywan commented 4 years ago

Managed to bust it again :D Being that it's a timeout error I've tested it over a number of days to be sure it's happening on the same user, and it is. User: jimjimjimmyjim

`Downloading thing 548 - 4124416 Traceback (most recent call last): File "D:\waste\thingygrabber\thingy_grabber.py", line 321, in _parse BROWSER.get(url) File "C:\Program Files\Python38\lib\site-packages\selenium\webdriver\remote\we bdriver.py", line 333, in get self.execute(Command.GET, {'url': url}) File "C:\Program Files\Python38\lib\site-packages\selenium\webdriver\remote\we bdriver.py", line 321, in execute self.error_handler.check_response(response) File "C:\Program Files\Python38\lib\site-packages\selenium\webdriver\remote\er rorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message: Timeout loading page after 300000ms

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "D:\waste\thingygrabber\thingy_grabber.py", line 690, in main() File "D:\waste\thingygrabber\thingy_grabber.py", line 676, in main Designs(user, args.directory, args.quick).download() File "D:\waste\thingygrabber\thingy_grabber.py", line 270, in download RC = Thing(thing).download(self.download_dir) File "D:\waste\thingygrabber\thingy_grabber.py", line 423, in download self._parse(base_dir) File "D:\waste\thingygrabber\thingy_grabber.py", line 330, in _parse logging.error(pc.log) UnboundLocalError: local variable 'pc' referenced before assignment `

Can not complete the user without this error occurring.

cwoac commented 4 years ago

OK. Weird. Is it always the same thing of that user?

cwoac commented 4 years ago

Hmmm. He has 1522 things. I can believe you are falling foul of rate limiting on the server side.

joebywan commented 4 years ago

I've downloaded things much faster than this in the past before, so it shouldn't be that.

On 28 Apr. 2020, 8:00 pm, at 8:00 pm, Oliver Matthews notifications@github.com wrote:

Hmmm. He has 1522 things. I can believe you are falling foul of rate limiting on the server side.

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/cwoac/thingy_grabber/issues/12#issuecomment-620506229

cwoac commented 4 years ago

So, the latest version removes the use of a background browser app, so there is no longer an asynchronous wait to timeout, so this is fixed by way of code deletion :).