Ignore robots by default

[jkaberg]

By default mechanize checks robots.txt before doing anything more, and since Plex is now using robots.txt we need to bypass this in order to make the container work. Obviously not what the Plex team wants

This fixes:

+ /retrieve_plex_download_url.py
Retrieving the latest Plex release for PlexPass users...
Traceback (most recent call last):
  File "/retrieve_plex_download_url.py", line 44, in <module>
    print(retrieve_latest_download_url(login, password))
  File "/retrieve_plex_download_url.py", line 21, in retrieve_latest_download_url
    browser.open('https://plex.tv/users/sign_in')
  File "/usr/lib/python2.7/dist-packages/mechanize/_mechanize.py", line 203, in open
    return self._mech_open(url, data, timeout=timeout)
  File "/usr/lib/python2.7/dist-packages/mechanize/_mechanize.py", line 255, in _mech_open
    raise response
mechanize._response.httperror_seek_wrapper: HTTP Error 403: request disallowed by robots.txt
+ curl -L -o plexmediaserver.deb
curl: /usr/lib/plexmediaserver/libcurl.so.4: no version information available (required by curl)
curl: no URL specified!
curl: try 'curl --help' or 'curl --manual' for more information

and (when applying ignore robots.txt patch you get:)

+ /retrieve_plex_download_url.py
Retrieving the latest Plex release for PlexPass users...
Traceback (most recent call last):
  File "/retrieve_plex_download_url.py", line 45, in <module>
    print(retrieve_latest_download_url(login, password))
  File "/retrieve_plex_download_url.py", line 22, in retrieve_latest_download_url
    browser.open('https://plex.tv/users/sign_in')
  File "/usr/lib/python2.7/dist-packages/mechanize/_mechanize.py", line 203, in open
    return self._mech_open(url, data, timeout=timeout)
  File "/usr/lib/python2.7/dist-packages/mechanize/_mechanize.py", line 230, in _mech_open
    response = UserAgentBase.open(self, request, data)
  File "/usr/lib/python2.7/dist-packages/mechanize/_opener.py", line 193, in open
    response = urlopen(self, req, data)
  File "/usr/lib/python2.7/dist-packages/mechanize/_urllib2_fork.py", line 344, in _open
    '_open', req)
  File "/usr/lib/python2.7/dist-packages/mechanize/_urllib2_fork.py", line 332, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/dist-packages/mechanize/_urllib2_fork.py", line 1170, in https_open
    return self.do_open(conn_factory, req)
  File "/usr/lib/python2.7/dist-packages/mechanize/_urllib2_fork.py", line 1118, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>
+ curl -L -o plexmediaserver.deb
curl: /usr/lib/plexmediaserver/libcurl.so.4: no version information available (required by curl)
curl: no URL specified!
curl: try 'curl --help' or 'curl --manual' for more information

[jkaberg]

Tested and confirmed working

[wernight]

I hit that issue as well. The weird thing is that robots.txt isn't disallowing any robot as far as I could tell. Also it happened only when reusing a container, and doing a docker-compose rm first fixes it for the next start.

Did you encounter any SSL issue? The Plex website should be signed and removing that check makes me uncomfortable. May be split it as another PR.

[jkaberg]

@wernight yeah I did get an ssl error (see OP), dont know why?

[wernight]

I may merge this as a temporary fix, but I'd like to find the root cause. Are you using docker or docker-compose and/or do you have steps to reproduce the issue?

[jkaberg]

Just normal docker, I simply run the container as in the readme.md and restart when I see an update (or like today when PLEX was acting up) which gave me the errors you see in OP

[wernight]

Steps then seem to be:

1. docker[-compose] up -d ...
2. wait (unknown duration, about days)
3. docker[-compose] stop && docker[-compose] up -d ...

Does that look like your steps? Did you for example just kill or did you docker rm it?

[jkaberg]

@wernight I only did docker restart <container id> (after initial setup)

[wernight]

Yes so that still follows the steps. docker rm also fixes it, so there must be something remaining, some caching. I'm disappointed that Mechanize isn't documenting properly its caching mechanism.