search

pytr-org / pytr

Use TradeRepublic in terminal and mass download all documents
https://pypi.org/project/pytr/
MIT License
420 stars 74 forks source link

Connection error while downloading documents #25

Closed Johannes0941 closed 2 years ago

Johannes0941 commented 2 years ago

Description of the bug While trying to download the documents with dl_docs the code stops in the middle

To Reproduce Hard to describe, as it is probably user dependent, my guess would be, that the pytr package might not support options trading and it canceled at the first of these documents. Maybe a better error handling strategy would be to to skip these documents and then name all skipped documents at the end of code execution

Error log

(base) C:\Users\49176\Documents\trade_republic>pytr dl_docs .
17:27:53 Found credentials file
17:27:53 Phone: +49176********, PIN: ****
17:27:54 Web session resumed
17:27:54 Logged in
17:27:54 Found 98 lines in history file
17:27:54 Awaiting #1  timeline
17:27:54 Connecting to websocket ...
17:27:54 Connected to websocket ...
17:27:54 Received #1  timeline, awaiting #2  timeline
17:27:55 Received #2  timeline, awaiting #3  timeline
17:27:55 Received #3  (last) timeline
17:27:55  1/65: Deine Unterlagen -- Sa. 30.10. um 21:41 Uhr (Europe/Berlin)
17:27:55  2/65: Basisinformationen -- Sa. 30.10. um 21:41 Uhr (Europe/Berlin)
17:27:55  3/65: Abschluss Q4/2021 -- Sa. 01.01. um 00:59 Uhr (Europe/Berlin)
17:27:55  4/65: Kauf Beyond Meat -- Mi. 05.01. um 20:59 Uhr (Europe/Berlin)
17:27:55  5/65: Core MSCI World USD (Acc) -- Sparplan
17:27:55  6/65: Core MSCI EM IMI USD (Acc) -- Sparplan
17:27:55  7/65: Kauf GoodYear -- Do. 03.03. um 14:22 Uhr (Europe/Berlin)
17:27:55  8/65: Kauf GoodYear -- Do. 03.03. um 15:44 Uhr (Europe/Berlin)
17:27:55  9/65: Kauf Global Clean Energy USD (Dist) -- Fr. 04.03. um 14:01 Uhr (Europe/Berlin)
17:27:55 10/65: Kauf BASF -- Do. 03.03. um 16:13 Uhr (Europe/Berlin)
17:27:55 11/65: Kauf ITM Power -- Fr. 04.03. um 14:34 Uhr (Europe/Berlin)
17:27:55 12/65: Kauf ITM Power -- Fr. 04.03. um 14:34 Uhr (Europe/Berlin)
17:27:55 13/65: Kauf Krones -- Fr. 04.03. um 14:38 Uhr (Europe/Berlin)
17:27:55 14/65: S&P Global Luxury EUR (Acc) -- Sparplan
17:27:55 15/65: Kauf BASF -- Fr. 04.03. um 14:41 Uhr (Europe/Berlin)
17:27:55 16/65: Kauf Call @41.00 € Bayer AG Optionsschein -- Di. 08.03. um 14:12 Uhr (Europe/Berlin)
17:27:55 17/65: Call @41.00 € Bayer AG Optionsschein -- Mi. 09.03. um 01:00 Uhr (Europe/Berlin)
17:27:55 18/65: Kauf Long @4.49 € ITM Power Open End Turbo -- Mi. 09.03. um 08:30 Uhr (Europe/Berlin)
17:27:55 19/65: Kauf Long @50.32 € BASF Open End Turbo -- Mi. 09.03. um 13:35 Uhr (Europe/Berlin)
17:27:55 20/65: Kauf Call @60.00 € BASF Optionsschein -- Mi. 09.03. um 15:13 Uhr (Europe/Berlin)
17:27:55 21/65: Call @60.00 € BASF Optionsschein -- Do. 10.03. um 00:22 Uhr (Europe/Berlin)
17:27:55 22/65: Kauf Core MSCI World USD (Acc) -- Do. 10.03. um 11:20 Uhr (Europe/Berlin)
17:27:55 23/65: Kauf Global Clean Energy USD (Dist) -- Do. 10.03. um 09:07 Uhr (Europe/Berlin)
17:27:55 24/65: Kauf Core MSCI EM IMI USD (Acc) -- Do. 10.03. um 11:24 Uhr (Europe/Berlin)
17:27:55 25/65: Edge Europe Value EUR (Acc) -- Sparplan
17:27:55 26/65: Kauf Upstart Holdings -- Do. 10.03. um 13:15 Uhr (Europe/Berlin)
17:27:55 27/65: Kauf Daimler Truck -- Do. 10.03. um 20:14 Uhr (Europe/Berlin)
17:27:55 28/65: Kauf Daimler Truck -- Do. 10.03. um 20:14 Uhr (Europe/Berlin)
17:27:55 29/65: Kauf Global Clean Energy USD (Dist) -- Do. 10.03. um 14:46 Uhr (Europe/Berlin)
17:27:55 30/65: Core MSCI EM IMI USD (Acc) -- Sparplan
17:27:56 31/65: Kauf BASF -- Do. 10.03. um 20:15 Uhr (Europe/Berlin)
17:27:56 32/65: Kauf Core MSCI World USD (Acc) -- Do. 10.03. um 20:55 Uhr (Europe/Berlin)
17:27:56 33/65: Kauf BASF -- Do. 10.03. um 20:16 Uhr (Europe/Berlin)
17:27:56 34/65: Kauf Core MSCI World USD (Acc) -- Do. 10.03. um 20:54 Uhr (Europe/Berlin)
17:27:56 35/65: Kauf Core MSCI World USD (Acc) -- Do. 10.03. um 20:55 Uhr (Europe/Berlin)
17:27:56 36/65: Kauf Shopify -- Do. 10.03. um 20:59 Uhr (Europe/Berlin)
17:27:56 37/65: Verkauf Long @4.49 € ITM Power Open End Turbo -- Fr. 11.03. um 00:12 Uhr (Europe/Berlin)
17:27:56 38/65: Kauf Shopify -- Fr. 11.03. um 11:10 Uhr (Europe/Berlin)
17:27:56 39/65: Kauf Shopify -- Fr. 11.03. um 12:52 Uhr (Europe/Berlin)
17:27:56 40/64: Verkauf Long @50.32 € BASF Open End Turbo -- Fr. 11.03. um 12:29 Uhr (Europe/Berlin)
17:27:56 41/64: Kauf Shopify -- Fr. 11.03. um 13:19 Uhr (Europe/Berlin)
17:27:56 42/64: Kauf Shopify -- Fr. 11.03. um 13:19 Uhr (Europe/Berlin)
17:27:56 43/64: Verkauf Call @41.00 € Bayer AG Optionsschein -- Fr. 11.03. um 15:38 Uhr (Europe/Berlin)
17:27:56 44/64: Kauf Edge Europe Value EUR (Acc) -- Fr. 11.03. um 15:48 Uhr (Europe/Berlin)
17:27:56 45/64: Kauf Edge Europe Value EUR (Acc) -- Fr. 11.03. um 15:47 Uhr (Europe/Berlin)
17:27:56 46/64: Verkauf Shopify -- Fr. 11.03. um 16:56 Uhr (Europe/Berlin)
17:27:56 47/64: Global Clean Energy USD (Dist) -- Sparplan
17:27:56 48/64: Verkauf Shopify -- Fr. 11.03. um 19:11 Uhr (Europe/Berlin)
17:27:56 49/64: Verkauf Shopify -- Fr. 11.03. um 19:12 Uhr (Europe/Berlin)
17:27:56 50/64: Kauf BASF -- Fr. 11.03. um 20:35 Uhr (Europe/Berlin)
17:27:56 51/64: Kauf Krones -- Fr. 11.03. um 20:35 Uhr (Europe/Berlin)
17:27:57 52/64: Dokumente akzeptiert -- Sa. 12.03. um 00:03 Uhr (Europe/Berlin)
17:27:57 53/64: Kauf Edge Europe Value EUR (Acc) -- Fr. 11.03. um 20:35 Uhr (Europe/Berlin)
17:27:57 54/64: Core STOXX Europe 600 EUR (Acc) -- Sparplan
17:27:57 55/64: Kauf Bitcoin -- Sa. 12.03. um 01:05 Uhr (Europe/Berlin)
17:27:57 56/64: FTSE EPRA Nareit Global EUR (Acc) -- Sparplan
17:27:57 57/64: MSCI World SRI EUR (Acc) -- Sparplan
17:27:57 58/64: Tilgung -- Mi. 16.03. um 01:26 Uhr (Europe/Berlin)
17:27:57 59/64: Kauf Covestro -- Di. 15.03. um 18:53 Uhr (Europe/Berlin)
17:27:57 60/64: Core MSCI World USD (Acc) -- Sparplan
17:27:57 61/64: Kauf Ares Capital -- Mi. 16.03. um 20:40 Uhr (Europe/Berlin)
17:27:57 62/64: Kauf Omega Healthcare Invs -- Fr. 18.03. um 14:18 Uhr (Europe/Berlin)
17:27:57 63/64: Verkauf Ares Capital -- Mi. 16.03. um 20:56 Uhr (Europe/Berlin)
17:27:57 64/64: Kauf Omega Healthcare Invs -- Fr. 18.03. um 14:39 Uhr (Europe/Berlin)
17:27:57 Received all details
17:27:57 Write deposit entries
17:27:57 Deposit creation finished!
17:27:57 Waiting for downloads to complete..
Traceback (most recent call last):
  File "C:\Users\49176\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\49176\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "C:\Users\49176\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\49176\anaconda3\lib\http\client.py", line 1371, in getresponse
    response.begin()
  File "C:\Users\49176\anaconda3\lib\http\client.py", line 319, in begin
    version, status, reason = self._read_status()
  File "C:\Users\49176\anaconda3\lib\http\client.py", line 288, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\49176\anaconda3\lib\site-packages\requests\adapters.py", line 439, in send
    resp = conn.urlopen(
  File "C:\Users\49176\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "C:\Users\49176\anaconda3\lib\site-packages\urllib3\util\retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Users\49176\anaconda3\lib\site-packages\urllib3\packages\six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "C:\Users\49176\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\49176\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "C:\Users\49176\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\49176\anaconda3\lib\http\client.py", line 1371, in getresponse
    response.begin()
  File "C:\Users\49176\anaconda3\lib\http\client.py", line 319, in begin
    version, status, reason = self._read_status()
  File "C:\Users\49176\anaconda3\lib\http\client.py", line 288, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\49176\anaconda3\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\49176\anaconda3\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\49176\anaconda3\Scripts\pytr.exe\__main__.py", line 7, in <module>
  File "C:\Users\49176\anaconda3\lib\site-packages\pytr\main.py", line 157, in main
    asyncio.get_event_loop().run_until_complete(dl.dl_loop())
  File "C:\Users\49176\anaconda3\lib\asyncio\base_events.py", line 642, in run_until_complete
    return future.result()
  File "C:\Users\49176\anaconda3\lib\site-packages\pytr\dl.py", line 64, in dl_loop
    await self.tl.timelineDetail(response, self, max_age_timestamp=self.since_timestamp)
  File "C:\Users\49176\anaconda3\lib\site-packages\pytr\utils.py", line 371, in timelineDetail
    dl.work_responses()
  File "C:\Users\49176\anaconda3\lib\site-packages\pytr\dl.py", line 149, in work_responses
    r = future.result()
  File "C:\Users\49176\anaconda3\lib\concurrent\futures\_base.py", line 438, in result
    return self.__get_result()
  File "C:\Users\49176\anaconda3\lib\concurrent\futures\_base.py", line 390, in __get_result
    raise self._exception
  File "C:\Users\49176\anaconda3\lib\concurrent\futures\thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\Users\49176\anaconda3\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\49176\anaconda3\lib\site-packages\requests\sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\49176\anaconda3\lib\site-packages\requests\adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Maybe one way to solve this is if you could tell me one line of code, where I could insert a print, to know after which iteration the code stops. Connection during code execution was always good, so I can´t imagine that to be the problem.

Thank you for your help and the very promising tool!

Johannes0941 commented 2 years ago

UPDATE: After multiple execution of the command the download of all documents is successfull, however this took about 5 iterations. However I don´t think that it is an error in the code but rather a problem of the API of Trade Republic maybe. Maybe you have more thougts on that, otherwise it can be closed.

marzzzello commented 2 years ago

Mh maybe there are too many parallel tcp connections. Which version do you use? Can you run the script in debug mode? pytr -v debug dl_docs... Or you wait until I have implemented some better error handling

marzzzello commented 2 years ago

@Johannes0941 I added a new option --workers. The default value was 8. Maybe a lower value helps for you. Also it will not fail anymore but log a error and continue

Johannes0941 commented 2 years ago

I used the version available 2 days ago. I created a new empty folder now and downloaded the documents in debug mode -> no error deleted all the documents and ran without debug mode -> also no error anymore So I wasnt´t able to reproduce the error or determine, if it was some specific document, probably my computer/ internet connection had some problems that day.

Also your Package is linked in the Parquet community forum, maybe more people there who can benefit from the package :) https://community.parqet.com/c/knowledge/trade-republic-abrechnungen-automatisiert