PromyLOPh / crocoite

Web archiving using Google Chrome
https://6xq.net/crocoite/
MIT License
42 stars 7 forks source link

Crashing when request is sent twice #24

Open PromyLOPh opened 4 years ago

PromyLOPh commented 4 years ago

It seems that Chrome sometime sends a request twice and then messes up the order in which it sends events, i.e. requestWillBeSent, requestWillBeSent (same id), loadingFinished. This results in the following crash:

Traceback (most recent call last):
  File "[…]/lib/python3.6/site-packages/crocoite/cli.py", line 136, in single
    loop.run_until_complete(run)
  File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "[…]/lib/python3.6/site-packages/crocoite/controller.py", line 259, in run
    handle.result ()
  File "[…]/lib/python3.6/site-packages/crocoite/controller.py", line 198, in processQueue
    async for item in l:
  File "[…]/lib/python3.6/site-packages/crocoite/browser.py", line 348, in __aiter__
    result = t.result ()
  File "[…]/lib/python3.6/site-packages/crocoite/browser.py", line 448, in _loadingFinished
    item.fromLoadingFinished (kwargs)
  File "[…]/lib/python3.6/site-packages/crocoite/browser.py", line 213, in fromLoadingFinished
    self.response.bytesReceived = data['encodedDataLength']
AttributeError: 'NoneType' object has no attribute 'bytesReceived'

This happens frequently on Twitter, when two iFrames load the same resource.