[bug] Error 406 when download source package

[madebr]

When (trying to) downloading the angelscript sdk at http://angelcode.com/angelscript/sdk/files/angelscript_2.34.0.zip using conan, I get a http error 406.

Using wget "http://angelcode.com/angelscript/sdk/files/angelscript_2.34.0.zip" works.

Environment Details (include every applicable attribute)

• Operating System+version: Fedora Linux 30
• Conan version: 1.28.1
• Python version: 3.7.7

logs

>>> import conans
>>> conans.tools.download(url="http://angelcode.com/angelscript/sdk/files/angelscript_2.34.0.zip", filename="angelscript_2.34.0.zip", retry=5, retry_wait=5)
ERROR: Error 406 downloading file http://angelcode.com/angelscript/sdk/files/angelscript_2.34.0.zip
Waiting 5 seconds to retry...
ERROR: Error 406 downloading file http://angelcode.com/angelscript/sdk/files/angelscript_2.34.0.zip
Waiting 5 seconds to retry...
ERROR: Error 406 downloading file http://angelcode.com/angelscript/sdk/files/angelscript_2.34.0.zip
Waiting 5 seconds to retry...
ERROR: Error 406 downloading file http://angelcode.com/angelscript/sdk/files/angelscript_2.34.0.zip
Waiting 5 seconds to retry...
ERROR: Error 406 downloading file http://angelcode.com/angelscript/sdk/files/angelscript_2.34.0.zip
Waiting 5 seconds to retry...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/maarten/.local/lib/python3.7/site-packages/conans/tools.py", line 78, in download
    return tools_net.download(out=_global_output, requester=_global_requester, *args, **kwargs)
  File "/home/maarten/.local/lib/python3.7/site-packages/conans/client/tools/net.py", line 114, in download
    _download_file(url)
  File "/home/maarten/.local/lib/python3.7/site-packages/conans/client/tools/net.py", line 104, in _download_file
    overwrite=overwrite, auth=auth, headers=headers)
  File "/home/maarten/.local/lib/python3.7/site-packages/conans/client/rest/file_downloader.py", line 45, in download
    headers, file_path)
  File "/home/maarten/.local/lib/python3.7/site-packages/conans/client/rest/file_downloader.py", line 150, in _call_with_retry
    return method(*args, **kwargs)
  File "/home/maarten/.local/lib/python3.7/site-packages/conans/client/rest/file_downloader.py", line 72, in _download_file
    raise ConanException("Error %d downloading file %s" % (response.status_code, url))
conans.errors.ConanException: Error 406 downloading file http://angelcode.com/angelscript/sdk/files/angelscript_2.34.0.zip

[lasote]

I reproduced the error. The server is blocking the request, I'm going to investigate a bit more.

[lasote]

Related to https://github.com/conan-io/conan/issues/7349 Changing the user-agent header works (Allowed thanks to https://github.com/conan-io/conan/pull/7390), the server is blocking the default from the python requests library:

tools.download("http://angelcode.com/angelscript/sdk/files/angelscript_2.34.0.zip", filename="angelscript_2.34.0.zip", headers={"User-Agent": "foo"})

[madebr]

Totally related. https://github.com/conan-io/conan/issues/7349#issuecomment-657875347 even used the same url :man_facepalming:

[Wuestenschiff]

I don't get it. How do I overwrite the user-agent when I calll conan-install?

[memsharded]

Hi @Wuestenschiff

You don't change the user-agent of Conan globally for all requests. A specific recipe can use a different agent if necessary in its tools.download() call in the source() method for example, if the server is picky about the agent. But this needs to be edited in the recipe.

[Wuestenschiff]

@memsharded Well then I understood but this is a problem when server just decide to block requests and brick builds. (like cloudlfare in https://github.com/conan-io/conan-center-index/issues/7478)

I think a possility to change this for clients would be very appreciated in such situations

[memsharded]

@lasote @jgsogo, it seems the python-requests user agent is the one generally having issues in some servers. Is it that necessary? Maybe we could just remove it?

[lasote]

I don't have any reason to keep it.

[jgsogo]

Hi! I'm not sure if this is related to the user-agent.... but if it fixes the issue, IMO it is safe to remove the python-requests part.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/406

The HyperText Transfer Protocol (HTTP) 406 Not Acceptable client error response code indicates that the server cannot produce a response matching the list of acceptable values defined in the request's proactive content negotiation headers, and that the server is unwilling to supply a default representation.

Proactive content negotiation headers include:

Accept Accept-Encoding Accept-Language

[lasote]

I created https://github.com/conan-io/conan/issues/9760 and assigned it to the release.

[madebr]

I have this problem when packaging arm toolchains at https://github.com/conan-io/conan-center-index/pull/6822, Adding the following http headers fixes it:

from conans import __version__ as conan_version
tools.download(**self._sources_dict, filename="archive.tar.xz",
               headers={"User-Agent": "Conan v{}".format(conan_version)})