TheMovieDB.org API Max Retries Exceeded

[SerinitySW]

Docker version 2.7.0 (Performance branch)

It's quite common for the script the crash for me with the following:

| 269 missing movies from IMDb List: https://www.imdb.com/list/ls076031090/
|
| Processing imdb_list: https://www.imdb.com/list/ls076077386/
| 369 Movies found on IMDb
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
conn = connection.create_connection(
File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 73, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/local/lib/python3.9/socket.py", line 953, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/usr/local/lib/python3.9/http/client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.9/http/client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.9/http/client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.9/http/client.py", line 1010, in _send_output
self.send(msg)
File "/usr/local/lib/python3.9/http/client.py", line 950, in send
self.connect()
File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 200, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 181, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x1465225c7520>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 573, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))

urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='api.themoviedb.org', port=80): Max retries exceeded with url: /3/find/tt0044837?api_key=XXXXXXXXXXXXXXXXXXXXXXXXXXXX&external_source=imdb_id&language=en (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1465225c7520>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/app/plex_auto_collections.py", line 1026, in <module>
update_from_config(config_path, plex, True, args.no_meta, args.no_images)
File "/app/plex_auto_collections.py", line 668, in update_from_config
missing, map = add_to_collection(config_path, plex, m, v, c, plex_map, map, filters)
File "/app/plex_tools.py", line 225, in add_to_collection
items, missing = imdb_tools.imdb_get_movies(config_path, plex, plex_map, value)
File "/app/imdb_tools.py", line 111, in imdb_get_movies
tmdb_id = imdb_get_tmdb(config_path, imdb_id)
File "/app/imdb_tools.py", line 82, in imdb_get_tmdb
search = movie.external(external_id=imdb_id, external_source="imdb_id")['movie_results']
File "/usr/local/lib/python3.9/site-packages/tmdbv3api/objs/movie.py", line 182, in external
self._call(
File "/usr/local/lib/python3.9/site-packages/tmdbv3api/tmdb.py", line 132, in _call
req = self.cached_request(method, url, data)
File "/usr/local/lib/python3.9/site-packages/tmdbv3api/tmdb.py", line 112, in cached_request
return requests.request(method, url, data=data)
File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='api.themoviedb.org', port=80): Max retries exceeded with url: /3/find/tt0044837?api_key=XXXXXXXXXXXXXXXXXXXXXXXXXXXX&external_source=imdb_id&language=en (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1465225c7520>: Failed to establish a new connection: [Errno -2] Name or service not known'))

API key is censored on purpose.

[twhiting9275]

This has become standard for me, making this script pretty worthless.. Constant connection and timeout issues, same exact errors

requests.exceptions.ConnectionError: HTTPConnectionPool(host='api.themoviedb.org', port=80): Max retries exceeded with url: /3/movie/1894?api_key=xxxxxxxxxx&append_to_response=videos,trailers,images,casts,translations,keywords&language=en (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x065C1FA0>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))

[mza921]

Is this still occurring? I'm hoping that the guid mapping in 2.8.1+ will cut back on the number of API requests against TMDB.

[SerinitySW]

Yeah, I can't really seem to get a complete run very often. Fails almost every time

[mza921]

Can you post the collection and lists where this is occurring?

[SerinitySW]

Absolutely. It happens randomly on random lists, but the last time it happened was on my oscar nominated collection:

      Oscar Nominated:
            imdb_list:
                - https://www.imdb.com/list/ls054248868/
                - https://www.imdb.com/list/ls024957857/
                - https://www.imdb.com/list/ls076034778/
                - https://www.imdb.com/list/ls076031090/
                - https://www.imdb.com/list/ls076077386/
                - https://www.imdb.com/list/ls073881227/
                - https://www.imdb.com/list/ls073322289/
                - https://www.imdb.com/list/ls073083957/
                - https://www.imdb.com/list/ls071960678/
                - https://www.imdb.com/list/ls071931345/

According to the logs, it happened on the 2nd list: https://ghostbin.com/paste/uhGsI

[mza921]

Is there anything in your network path that could be throttling your connection or are you using a VPN? socket.gaierror: [Errno -2] Name or service not known is such a low-level error and it's odd that it would be random.

[burkasaurusrex]

Definitely a very strange error, especially since TheMovieDB doesn’t rate limit. @mza921 I think to get to the bottom of this, we may need to enable the debug mode in tmdbv3api with tmdb.debug = True

[burkasaurusrex]

Also weird that tmdbv3api is trying to connect to api.themoviedb.com on port 80 instead of port 443

[mza921]

Try running the docker container with -e TMDB_DEBUG_ENABLED=True

[burkasaurusrex]

https://github.com/AnthonyBloomer/tmdbv3api/blob/514e37d81d4aa399fb73de65edc00c465205cd54/tmdbv3api/tmdb.py#L30

tvdbv3api is pointing to an http endpoint instead of an https endpoint. My working hypothesis is that some environments are failing to get the redirect to https on occasion.

[mza921]

I think I can also alleviate some of these external API hits by having it check the guid map before hitting the TMDB API.

[burkasaurusrex]

It looks like tmdbv3api also has a cache that might be worth enabling

[burkasaurusrex]

I’m hoping that changing the HTTP endpoint to an HTTPS endpoint will fix this though. I just opened an issue on that repo as well.

[burkasaurusrex]

Also, I haven’t looked at the code in a while, but last I looked it wasn’t reusing the same tmdbv3api object? It was reinstantiating the object for every list? If that’s still the case, it would dramatically cut down calls just by reusing the same object to avoid reauthenticating every time.

[mza921]

Looks like it reinstantiates for every item.

[SerinitySW]

Is there anything in your network path that could be throttling your connection or are you using a VPN?

I'm not entirely sure, I'm not super network-savvy. I do know the script and Plex are not behind a VPN however. My home router is running pfSense if that helps.

Try running the docker container with -e TMDB_DEBUG_ENABLED=True

Here you go: https://ghostbin.com/paste/S9qyl

[Tharic99]

Seeing the same errors here today trying to add a ton of new collections to my config and re-running the script a few dozen times before it eventually just dies on me.

Does it eventually come back in 24 hours or something?

[Tharic99]

@burkasaurusrex

Definitely a very strange error, especially since TheMovieDB doesn’t rate limit.

https://www.themoviedb.org/talk/5317af69c3a3685c4a0003b1?language=en-US

[burkasaurusrex]

Version 1.7.3 of tmdbv3api dropped last week with the HTTPS fix ... will submit a PR to bump the version, but if anybody is experiencing this problem, feel free to adjust the requirements.txt and test. Thanks.

[burkasaurusrex]

Version 2.9.0 has the fix to the HTTPS issue mentioned above. Do you all mind retesting to see if that helped? If not, we'll need to do a lot more digging I think

[SerinitySW]

Retested, getting the same error at about the same frequency. v2.9.0