[BUG] Timeouts may be too low in `aiohttp.ClientSession`

[brettpetch]

Describe the bug Timeouts are too low for large libraries.

To Reproduce Steps to reproduce the behavior: Run on a rather large library.

Expected behavior The application moves watch history in its entirety, has a configurable timeout for aiohttp.ClientSession.

Logs If applicable, add logs to help explain your problem ideally with DEBUG set to true, be sure to remove sensitive information

Jellyfin: Generating watched for username in library TV (Sports)
[ERROR]: Jellyfin: Query get /Users/ID/Items?ParentId=ID&isPlaceHolder=false&IncludeItemTypes=Series&Recursive=True&Fields=ProviderIds,Path,RecursiveItemCount
Results None

[ERROR]: Jellyfin: Failed to get watched for username in library TV, Error: 
[ERROR]: Traceback (most recent call last):
  File "/opt/jf-watched/JellyPlex-Watched/src/jellyfin.py", line 96, in query
    async with session.get(
  File "/opt/jf-watched/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "/opt/jf-watched/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 560, in _request
    await resp.start(conn)
  File "/opt/jf-watched/.venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 894, in start
    with self._timer:
  File "/opt/jf-watched/.venv/lib/python3.11/site-packages/aiohttp/helpers.py", line 721, in __exit__
    raise asyncio.TimeoutError from None
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/jf-watched/JellyPlex-Watched/src/jellyfin.py", line 221, in get_user_library_watched
    watched_shows = await self.query(
                    ^^^^^^^^^^^^^^^^^
  File "/opt/jf-watched/JellyPlex-Watched/src/jellyfin.py", line 126, in query
    raise Exception(e)
Exception

[ERROR]: Jellyfin: Query get /Users/ID/Items?ParentId=ID&isPlaceHolder=false&IncludeItemTypes=Series&Recursive=True&Fields=ProviderIds,Path,RecursiveItemCount
Results None

[ERROR]: Jellyfin: Failed to get watched for username in library TV (Foreign), Error: 
[ERROR]: Traceback (most recent call last):
  File "/opt/jf-watched/JellyPlex-Watched/src/jellyfin.py", line 96, in query
    async with session.get(
  File "/opt/jf-watched/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "/opt/jf-watched/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 560, in _request
    await resp.start(conn)
  File "/opt/jf-watched/.venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 894, in start
    with self._timer:
  File "/opt/jf-watched/.venv/lib/python3.11/site-packages/aiohttp/helpers.py", line 721, in __exit__
    raise asyncio.TimeoutError from None
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/jf-watched/JellyPlex-Watched/src/jellyfin.py", line 221, in get_user_library_watched
    watched_shows = await self.query(
                    ^^^^^^^^^^^^^^^^^
  File "/opt/jf-watched/JellyPlex-Watched/src/jellyfin.py", line 126, in query
    raise Exception(e)
Exception

[ERROR]: Jellyfin: Query get /Users/ID/Items?ParentId=ID&isPlaceHolder=false&IncludeItemTypes=Series&Recursive=True&Fields=ProviderIds,Path,RecursiveItemCount
Results None

[ERROR]: Jellyfin: Failed to get watched for username in library TV, Error: 
[ERROR]: Traceback (most recent call last):
  File "/opt/jf-watched/JellyPlex-Watched/src/jellyfin.py", line 96, in query
    async with session.get(
  File "/opt/jf-watched/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "/opt/jf-watched/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 560, in _request
    await resp.start(conn)
  File "/opt/jf-watched/.venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 894, in start
    with self._timer:
  File "/opt/jf-watched/.venv/lib/python3.11/site-packages/aiohttp/helpers.py", line 721, in __exit__
    raise asyncio.TimeoutError from None
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/jf-watched/JellyPlex-Watched/src/jellyfin.py", line 221, in get_user_library_watched
    watched_shows = await self.query(
                    ^^^^^^^^^^^^^^^^^
  File "/opt/jf-watched/JellyPlex-Watched/src/jellyfin.py", line 126, in query
    raise Exception(e)
Exception

Type:

• [x] Docker
• [x] Native

Additional context Add any other context about the problem here.

[NotMattch]

Same issue here, errors are identical to the ones posted in OP

[luigi311]

That's interesting, I haven't ran into this with my libraries though maybe mine aren't as big as your guys. It sounds like it would be best for me to set that timeout as a variable with a higher default that way if its still not high enough you guys can increase it on your side.

[NotMattch]

That's interesting, I haven't ran into this with my libraries though maybe mine aren't as big as your guys. It sounds like it would be best for me to set that timeout as a variable with a higher default that way if its still not high enough you guys can increase it on your side.

That sounds great! It's only 2 of my libraries that it will time out on, which are about 8TB and 10TB large, the ones it runs fine on are all sub 3TB

[Webreaper]

Also seeing timeouts syncing between Plex and JF. I'm running docker.

If it helps, my TV lib has around 17,000 episodes, 14TB. Configurable timeout would be great!

PS: While I'm here, thanks for this awesome tool. I was using the JF recommended migration tool and it didn't work at all...

[brettpetch]

Also seeing timeouts syncing between Plex and JF. I'm running docker.

If it helps, my TV lib has around 17,000 episodes, 14TB. Configurable timeout would be great!

PS: While I'm here, thanks for this awesome tool. I was using the JF recommended migration tool and it didn't work at all...

May also be worth noting that on large libs like this, sometimes the Jellyfin DB itself will OOM the server, so a configurable delay between requests may also be desirable.

[luigi311]

Theres now a timeout branch/tag, can you guys give it a test and see if you are able to adjust the REQUEST_TIMEOUT variable and increase it to something that works for you. I set the default time for total requests to 300 seconds and disabled the individual timeouts for things like connection and read requests and feel free to increase the timeout time as much as needed.

[Webreaper]

Is that branch available via docker? I tried to pull luigi311/jellyplex-watched:timeout but it wasn't recognised.

If you haven't pushed an image, I'll wait and try it out later.

[luigi311]

Sorry the tag is timeout-alpine or timeout-slim, i should probably adjust he workflow so it creates a version of just the branch name similar to latest.

[Webreaper]

Perfect, thanks. Will try it now.

[Webreaper]

Set it to 600s and it appears to now be churning through the library nicely - thanks for the fix!

[luigi311]

@Webreaper Thanks for the test, i pushed the fix to the main branch so latest should now work