from pypdl import Pypdl
from time import sleep
download_list = [
"https://safebooru.org//images/4619/3eb0ebb8b3a93515fa070f6be303527c48ffeed1.jpg",
"https://safebooru.org//images/4619/55cdaa511197791cc818d0e9388e9f93afdd4c0d.jpg"
]
dl = Pypdl(allow_reuse=True, headers={
'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0"
})
for download in download_list:
dl.start(download, block=True, segments=2)
while dl.wait:
print(f"Waiting for download from {download}...")
sleep(0.2)
while not dl.completed:
print(f"Downloading from {download}...")
sleep(0.2)
The output gets stuck at Downloading from https://safebooru.org//images/4619/55cdaa511197791cc818d0e9388e9f93afdd4c0d.jpg....
I set block=True to demonstrate that the downloader broke at the second download. The first completes normally.
The default error log message is lacking, however when using extensions from PR #22 , the message logged is:
(Pypdl) 21-07-24 15:21:29 - ERROR: (AttributeError) ['NoneType' object has no attribute 'get']
Traceback (most recent call last):
File "C:\Users\Speyedr\Documents\PyCharm Projects\pypdl-sticky-header-investigation\pypdl\pypdl_manager.py", line 103, in download
result = self._execute(
^^^^^^^^^^^^^^
File "C:\Users\Speyedr\Documents\PyCharm Projects\pypdl-sticky-header-investigation\pypdl\pypdl_manager.py", line 175, in _execute
file_path, multisegment, etag = self._get_info(
^^^^^^^^^^^^^^^
File "C:\Users\Speyedr\Documents\PyCharm Projects\pypdl-sticky-header-investigation\pypdl\pypdl_manager.py", line 230, in _get_info
file_path = get_filepath(url, header, file_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Speyedr\Documents\PyCharm Projects\pypdl-sticky-header-investigation\pypdl\utls.py", line 32, in get_filepath
content_disposition = headers.get("Content-Disposition", None)
^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'
Further enhancing the debug log messages demonstrates what is being communicated between the client and server (from this commit):
Looking closely, we can see that on the second request, the client includes 'range': 'bytes=469060-938119' in the request header. This should not exist and is a remnant of a previous GET request header when downloading the first file.
This additional range header is why the server responds with 206 instead of 200, which agrees with MDN HTTP docs.
Looking closely at the conditions for Pypdl._get_header(), we see that it has no handling for 206 responses, which is why the function returns None and later generates an exception. If the server had simply ignored the range header and returned status 200, the client would have continued as normal, which is likely what many servers end up doing. Additionally, aiohttp docs state that raise_for_status=True will only raise an exception if the response is greater than 400, which explains why the 206 response fell through.
Code (bug_replication.py):
The output gets stuck at
Downloading from https://safebooru.org//images/4619/55cdaa511197791cc818d0e9388e9f93afdd4c0d.jpg...
. I setblock=True
to demonstrate that the downloader broke at the second download. The first completes normally.The default error log message is lacking, however when using extensions from PR #22 , the message logged is:
Further enhancing the debug log messages demonstrates what is being communicated between the client and server (from this commit):
Looking closely, we can see that on the second request, the client includes
'range': 'bytes=469060-938119'
in the request header. This should not exist and is a remnant of a previous GET request header when downloading the first file. This additionalrange
header is why the server responds with206
instead of200
, which agrees with MDN HTTP docs.Looking closely at the conditions for
Pypdl._get_header()
, we see that it has no handling for206
responses, which is why the function returnsNone
and later generates an exception. If the server had simply ignored therange
header and returned status200
, the client would have continued as normal, which is likely what many servers end up doing. Additionally, aiohttp docs state thatraise_for_status=True
will only raise an exception if the response is greater than400
, which explains why the206
response fell through.https://github.com/mjishnu/pypdl/blob/12a20ad6285fb0f4fddcb2ff81742b646c142c6a/pypdl/pypdl_manager.py#L247-L257