A simple workaround for a bug with a complicated chain of events, due an oversight in the usage of the Pypdl._kwargs attribute.
This PR fixes #23.
When creating a Pypdl object, one has the ability to pass in arbitrary headers which will then be forwarded later to an aiohttp.ClientSession() object. These headers are collected within **kwargs and then stored in the self._kwargs attribute:
In order to request ranges of bytes to perform multi-segment downloads, Multidown.worker() collects the user-supplied arguments in **kwargs and then later adds a range header, as this needs to be passed later to another aiohttp.ClientSession() object:
If a user has passed custom headers through Pypdl(headers=headers), then Line 65 permanently alters Pypdl._kwargs due to Python passing Pypdl._kwargs by reference instead of by value.
When a subsequent request is made later by Pypdl._get_header() (see Line 249 above), the range header is still present and will be included in a request that is not meant to include a range header.
Pypdl._get_header() only expects to receive status code 200 in response, however the MDN web docs suggest that servers should respond with 206 if a range header is supplied.
If a server does respond with 206 then the Pypdl._get_header() method falls off and returns None. This has dire consequences when used by Pypdl._get_info(), as it sets the local attribute header to None...
(Pypdl) 21-07-24 15:21:29 - ERROR: (AttributeError) ['NoneType' object has no attribute 'get']
Traceback (most recent call last):
File "C:\Users\Speyedr\Documents\PyCharm Projects\pypdl-sticky-header-investigation\pypdl\pypdl_manager.py", line 103, in download
result = self._execute(
^^^^^^^^^^^^^^
File "C:\Users\Speyedr\Documents\PyCharm Projects\pypdl-sticky-header-investigation\pypdl\pypdl_manager.py", line 175, in _execute
file_path, multisegment, etag = self._get_info(
^^^^^^^^^^^^^^^
File "C:\Users\Speyedr\Documents\PyCharm Projects\pypdl-sticky-header-investigation\pypdl\pypdl_manager.py", line 230, in _get_info
file_path = get_filepath(url, header, file_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Speyedr\Documents\PyCharm Projects\pypdl-sticky-header-investigation\pypdl\utls.py", line 32, in get_filepath
content_disposition = headers.get("Content-Disposition", None)
^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'
This bug has likely gone unsighted because it requires:
A user downloading multiple files using the same Pypdl object
The user passing in custom headers
The server supporting multi-segment downloads
The server returning 206 because the range header was included, instead of ignoring it and simply returning 200.
This PR includes a simple addition to the Multidown.worker() method that creates a shallow copy of kwargs['headers'] if it already exists, which disconnects it from the original reference and allows the range header to be added without affecting Pypdl._kwargs['header'].
This fix is faster than alternative patches which may require the use of deepcopy to properly disconnect or reset the Pypdl._kwargs attribute, however as a consequence this PR does not prevent the same issue from occurring again (i.e. unintentionally modifying attributes of Pypdl that have been passed by reference to callee functions).
A simple workaround for a bug with a complicated chain of events, due an oversight in the usage of the
Pypdl._kwargs
attribute.This PR fixes #23.
When creating a
Pypdl
object, one has the ability to pass in arbitrary headers which will then be forwarded later to anaiohttp.ClientSession()
object. These headers are collected within**kwargs
and then stored in theself._kwargs
attribute:https://github.com/mjishnu/pypdl/blob/12a20ad6285fb0f4fddcb2ff81742b646c142c6a/pypdl/pypdl_manager.py#L44-L48
This dictionary is passed to multiple objects during normal procedure, such as during the
Pypdl._get_header()
method...https://github.com/mjishnu/pypdl/blob/12a20ad6285fb0f4fddcb2ff81742b646c142c6a/pypdl/pypdl_manager.py#L247-L252
...and the
Pypdl._multi_segment()
method:https://github.com/mjishnu/pypdl/blob/12a20ad6285fb0f4fddcb2ff81742b646c142c6a/pypdl/pypdl_manager.py#L259-L270
In order to request ranges of bytes to perform multi-segment downloads,
Multidown.worker()
collects the user-supplied arguments in**kwargs
and then later adds arange
header, as this needs to be passed later to anotheraiohttp.ClientSession()
object:https://github.com/mjishnu/pypdl/blob/12a20ad6285fb0f4fddcb2ff81742b646c142c6a/pypdl/downloader.py#L46-L48 Line 65 updates the
kwargs['headers']
dictionary. https://github.com/mjishnu/pypdl/blob/12a20ad6285fb0f4fddcb2ff81742b646c142c6a/pypdl/downloader.py#L63-L66This has an unintended side-effect that:
Pypdl(headers=headers)
, then Line 65 permanently altersPypdl._kwargs
due to Python passingPypdl._kwargs
by reference instead of by value.Pypdl._get_header()
(see Line 249 above), therange
header is still present and will be included in a request that is not meant to include arange
header.Pypdl._get_header()
only expects to receive status code200
in response, however the MDN web docs suggest that servers should respond with206
if arange
header is supplied.206
then thePypdl._get_header()
method falls off and returnsNone
. This has dire consequences when used byPypdl._get_info()
, as it sets the local attributeheader
toNone
...https://github.com/mjishnu/pypdl/blob/12a20ad6285fb0f4fddcb2ff81742b646c142c6a/pypdl/pypdl_manager.py#L228-L230
get_filepath()
will result in aNone
reference error.https://github.com/mjishnu/pypdl/blob/12a20ad6285fb0f4fddcb2ff81742b646c142c6a/pypdl/utls.py#L31-L32
Stacktrace:
This bug has likely gone unsighted because it requires:
206
because therange
header was included, instead of ignoring it and simply returning200
.This PR includes a simple addition to the
Multidown.worker()
method that creates a shallow copy ofkwargs['headers']
if it already exists, which disconnects it from the original reference and allows therange
header to be added without affectingPypdl._kwargs['header']
.This fix is faster than alternative patches which may require the use of
deepcopy
to properly disconnect or reset thePypdl._kwargs
attribute, however as a consequence this PR does not prevent the same issue from occurring again (i.e. unintentionally modifying attributes ofPypdl
that have been passed by reference to callee functions).