Currently, the CKAN database stores resource URLs in percent-encoded form. When the archiver extension attempts to download a file using these URLs, there's an unintended double percent-encoding that can lead to download errors.
The problematic code responsible for this issue is located in the tidy_url() function within the tasks.py file:
# ckanext/archiver/tasks.py:653
# Find out if it has unicode characters, and if it does, quote them
# so we are left with an ASCII string
try:
url = url.decode('ascii')
except Exception:
parts = list(urlparse(url))
parts[2] = quote(parts[2].encode('utf-8'))
url = urlunparse(parts)
url = str(url)
In Python 3, the attempt to decode the URL into ASCII is unnecessary and, in fact, causes the except block to be applied resulting in the double encoding issue. Therefore, the entire try-except block for ASCII handling can be safely removed in Python 3 environments.
Furthermore, it's crucial to prioritize upgrading the archiver extension to Python 3 as soon as possible to eliminate other py2 issues and ensure compatibility.
Issue Description:
Currently, the CKAN database stores resource URLs in percent-encoded form. When the archiver extension attempts to download a file using these URLs, there's an unintended double percent-encoding that can lead to download errors.
The problematic code responsible for this issue is located in the
tidy_url()function within thetasks.pyfile:In Python 3, the attempt to decode the URL into ASCII is unnecessary and, in fact, causes the except block to be applied resulting in the double encoding issue. Therefore, the entire try-except block for ASCII handling can be safely removed in Python 3 environments.
Furthermore, it's crucial to prioritize upgrading the archiver extension to Python 3 as soon as possible to eliminate other py2 issues and ensure compatibility.