search

pawamoy / aria2p

Command-line tool and library to interact with an aria2c daemon process with JSON-RPC.
https://pawamoy.github.io/aria2p
ISC License
490 stars 93 forks source link

Remote end closed connection without response #130

Open Nubuki-all opened 10 months ago

Nubuki-all commented 10 months ago

Describe the bug aria2p suddenly hung up and disconnects after adding torrent with many files aria2c daemon process is also stopped after this The bug doesn't occur if torrent were added directly with aria2c.

To Reproduce Steps to reproduce the behavior:

  1. using aria2p as a library add torrent uri or magnetic link with aria2p.add
  2. If magnetic link error happens after getting metadata
  3. If torrent link error occurs after downloading torrent file

Expected behavior Download should proceed normally

Screenshots If applicable, add screenshots to help explain your problem.

System (please complete the following information):

Additional context Error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 461, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/http/client.py", line 1378, in getresponse
    response.begin()
  File "/usr/lib64/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 461, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/http/client.py", line 1378, in getresponse
    response.begin()
  File "/usr/lib64/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/bot/bot/workers/downloaders/download.py", line 373, in progress_for_aria2
    download = await sync_to_async(self.aria2.get_download, gid)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/bot/bot/utils/bot_utils.py", line 635, in sync_to_async
    return await future if wait else future
           ^^^^^^^^^^^^
  File "/usr/lib64/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aria2p/api.py", line 287, in get_download
    return Download(self, self.client.tell_status(gid))
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aria2p/client.py", line 880, in tell_status
    return self.call(self.TELL_STATUS, [gid, keys])  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aria2p/client.py", line 265, in call
    return self.res_or_raise(self.post(payload))
                             ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aria2p/client.py", line 361, in post
    return requests.post(self.server, data=payload, timeout=self.timeout).json()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
pawamoy commented 9 months ago

Hello, thanks for the report.

Does it occur reliably, or is this issue sporadic? I'm unsure how and why the client would crash the server itself. I'll try to replicate anyway.

hannesweisbach commented 7 months ago

Hello,

I think I experience the same issue. I'm running aria2c 1.36.0, aria2p 0.12.0 and python 3.11.6. Trying to add a torrent file results in a similar backtrace:

aria2p -H http://<ip> -s <mysecret> add-torrents  ~/Downloads/NMT.torrent 
Traceback (most recent call last):
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/urllib3/connectionpool.py", line 537, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/urllib3/connection.py", line 466, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 1378, in getresponse
    response.begin()
  File "/usr/lib/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/urllib3/connectionpool.py", line 537, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/urllib3/connection.py", line 466, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 1378, in getresponse
    response.begin()
  File "/usr/lib/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hannesweisbach/.local/venvs/aria2p/bin/aria2p", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/aria2p/cli/main.py", line 115, in main
    return commands[subcommand](api, **kwargs)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/aria2p/cli/commands/add_torrent.py", line 48, in add_torrents
    new_download = api.add_torrent(torrent_file, options=options, position=position)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/aria2p/api.py", line 151, in add_torrent
    gid = self.client.add_torrent(encoded_contents, uris, client_options, position)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/aria2p/client.py", line 528, in add_torrent
    return self.call(self.ADD_TORRENT, [torrent, uris, options, position])  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/aria2p/client.py", line 258, in call
    return self.res_or_raise(self.post(payload))
                             ^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/aria2p/client.py", line 351, in post
    return requests.post(self.server, data=payload, timeout=self.timeout).json()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

I was never able to add this torrent file at all. So as far as I can tell, it's always reproducible. I cannot rule out that the torrent file is maliciously crafted or that the error is with aria2c. But I thought I'd let you know, since it seemed relevant.

pawamoy commented 7 months ago

Thanks! Have you tried passing this file directly to aria2c? Does it accept it without error?

Nubuki-all commented 6 months ago

Hello, thanks for the report.

Does it occur reliably, or is this issue sporadic? I'm unsure how and why the client would crash the server itself. I'll try to replicate anyway.

Sorry I didn't realize this has been replied to It occurs reliably especially on certain torrents

pawamoy commented 6 months ago

Thanks, and no worries. I don't think I'll have time to investigate why a particular torrent makes it fail and how to fix that, but we can definitely improve the error handling here :slightly_smiling_face:

by-justin commented 1 month ago

If a torrent is too large (larger than 2M) , aria2c drops rpc connection.

Document says cranking this value up will solve the issue. I tried to set "rpc-max-request-size": "1073741824" in global options, it doesn't work, but other options do work.

So in the mean time, the simplest work around would be 

aria2c --enable-rpc --daemon=true --rpc-max-request-size=1024M
hannesweisbach commented 1 month ago

Thanks! Have you tried passing this file directly to aria2c? Does it accept it without error?

Sorry for the late reply. I did test it with aria2c directly and I did not observe aria2c throwing any errors nor did I see any errors in the log file.