Adding User-Agent string to urllib

[mikerenfro]

shellshare.net doesn't like urllib downloads of script.exe by default. Adding a User-Agent header to sidestep this until a server-side fix can be applied.

[vitorbaptista]

Hey @mikerenfro, when I try your patch, I get another error:

Traceback (most recent call last):
  File "shellshare", line 188, in <module>
    url_req.urlretrieve(script_url, script_path)
  File "/home/vitor/.pyenv/versions/3.7.7/lib/python3.7/urllib/request.py", line 247, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/home/vitor/.pyenv/versions/3.7.7/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/home/vitor/.pyenv/versions/3.7.7/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/home/vitor/.pyenv/versions/3.7.7/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/home/vitor/.pyenv/versions/3.7.7/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/home/vitor/.pyenv/versions/3.7.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/home/vitor/.pyenv/versions/3.7.7/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 520: Origin Error

I'm using Ubuntu 20.04. I modified the if platform.system() == 'Windows' to be if platform.system() != 'Windows' so I could test the code. Have you seen this?

[mikerenfro]

I haven't seen that before. Building some test code now.

[mikerenfro]

Test code, I think it's something server-side or network-related, as I've run this repeatedly with no changes to the Python code, and get the 520 error about 10% of the time.

#!/usr/bin/env python

try:
    import urllib.request as url_req  # for Python 3
    test_opener = True
    if test_opener:
        opener = url_req.build_opener()
        opener.addheaders = [('User-Agent', 'Mozilla/5.0')]
        url_req.install_opener(opener)
except ImportError:
    import urllib as url_req  # for Python 2
import os

bindir = os.path.join(os.path.expanduser('~'), '.shellshare')
if not os.path.exists(bindir):
    os.mkdir(bindir)
script_path = os.path.join(bindir, 'script.exe')
script_url = 'https://shellshare.net/bin/script.exe'
url_req.urlretrieve(script_url, script_path)

[mikerenfro]

@vitorbaptista so I guess see if there are any corresponding 520 errors logged in whatever stack of web servers run shellshare.net?