pytr dl_docs shows new documents but does not download them

[Lebensform123]

Description of the bug Starting this week (June 10th 2024) new documents are not downloaded autmatically by "pytr dl_docs". Till last friday (June 7th 2024) it worked without any issues.

To get more information I tried "pytr dl_docs -v debug" and I see that for the new messages the response from the server seems to look different. Before it looked like this:

Now the only response that there is is:

Can anybody reproduce the issue. Do you have any idea where this comes from? I do not see what I do different then before. Did something change in the API?

The issue ini the end is that the new files are not being downloaded.

Environment

• OS: [Windows 10]
• pytr version: [e.g. 0.1.9,

[tcdev0]

i got the following error since yesterday (pytr 0.1.9 on macOS 14.5)

pytr -v debug dl_docs ./docs
...
2024-06-11 13:35:53+0200 api       INFO     Connected to websocket ...
2024-06-11 13:35:53+0200 api       DEBUG    Subscribing: 'sub 1 {"type": "timeline", "after": null}'
2024-06-11 13:35:53+0200 api       DEBUG    Received message: '1 E {"errors":[{"errorCode":"UNKNOWN_ERROR","errorField":null,"errorMessage":"Unknown error","meta":{"source":"PLATFORM"}}]}'
2024-06-11 13:35:53+0200 api       ERROR    Received error message: '1 E {"errors":[{"errorCode":"UNKNOWN_ERROR","errorField":null,"errorMessage":"Unknown error","meta":{"source":"PLATFORM"}}]}'
2024-06-11 13:35:53+0200 api       DEBUG    Unsubscribing: 1
2024-06-11 13:35:53+0200 dl        CRITICAL ('1', {'type': 'timeline', 'after': None}, {'errors': [{'errorCode': 'UNKNOWN_ERROR', 'errorField': None, 'errorMessage': 'Unknown error', 'meta': {'source': 'PLATFORM'}}]})
Traceback (most recent call last):
  File "/Users/tcdev0/.local/bin/pytr", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/tcdev0/.local/pipx/venvs/pytr/lib/python3.12/site-packages/pytr/main.py", line 208, in main
    asyncio.get_event_loop().run_until_complete(dl.dl_loop())
  File "/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/tcdev0/.local/pipx/venvs/pytr/lib/python3.12/site-packages/pytr/dl.py", line 77, in dl_loop
    if subscription['type'] == 'timeline':
       ^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'subscription' where it is not associated with a value

[fft1010]

Hi, I can confirm, that no documents are downloaded since monday. Last DL was 8th june 2024. I just checked the file date. (Did not use it on the 9th although.)

pytr Version 0.1.9

thanks Juergen

[moeff]

TR changed something... hope someone fixes pytr soon, love this tool :)

[fft1010]

Hi, I have another program called "FinFUX" (a browser Extension). There it is the same. This program can download "Rechnungen" but not the "Abrechnungen" (in Abrechnungen there seem to be the final invoicing) any more since 10th june. SO IMHO defintely TR changed something.

I just noticed, that this program still can DL die PDF of "Rechnung" before june 10th, and everything else, but starting on june 10th it can DL everything except the "Rechnung". No idea what changed. I viewed the DL links, and they totally look the same.

thanks Juergen

[matawed]

Hi,

I also have that issue that pdfs are not downloaded for

pytr -v debug dl_docs --last_days 22 docs

I see some entries like

08:46:22 xx/yy: American Tower -- Sparplan

but no pdfs.

However, if I also checked my stock saving plans (Sparplan) in TR app (iOS) and can't find any recent "Abrechnung Ausführung" document in those plans for this month. Only for last month (and before). My assumption would be that those are not yet available, might be related to current technical issues/problems of TR :man_shrugging:

[fft1010]

Hi @matawed , the "Abrechungen" do exist in the TR webinterface. (at least here in my system), but still pytr does not download them.

The ones before 8th june can be DL by pytr, the ones since 10. june cannot be DL by pytr. In the TR web everything is there (manual work, we should invoice TR :-) )

thanks Juergen

[Lebensform123]

I had a look at the pytr-code (without really understanding all the details)

What I found is that the "recv()-function" of the TR-API does not provide all the "sections" anymore. Before June 10th it looked liked this:

now it looks like this:

As you can see the sections part is empty.

[fft1010]

Hi @Lebensform123 , thanks for digging in. As I see the "id" above, why not "simply" downloading it, and "hope" to get a PDF? For me it would be a great help, if I could get the PDFs automated, as I than load them into PortfolioPerformance.

Sorry, I have no clue in programming python. If you tell my how to get the Data from TR, I could look into my data, and maybe I can find some hints.

It seems, we somehow have to revers engeneer this. How hard would it be to brute force download the PDF?

thanks Juergen

[tcdev0]

after playing around, while not having great success i noticed something:

If you open https://app.traderepublic.com/profile/transactions and open the browser network tab there is an entry for the websocket data api.traderepublic.com

In api.py we subscribe to 'type': 'timeline' but i couldnt find this type but instead there was 'type': 'timelineTransactions', and timelineDetailV2 instead of timelineDetail.

after changing the parts, there was some data to recieve.

# api.py
...
async def timeline(self, after=None):
    #return await self.subscribe({'type': 'timeline', 'after': after})
    return await self.subscribe({'type': 'timelineTransactions', 'after': after})

async def timeline_detail(self, timeline_id):
    #return await self.subscribe({'type': 'timelineDetail', 'id': timeline_id})
    return await self.subscribe({'type': 'timelineDetailV2', 'id': timeline_id})

async def timeline_detail_order(self, order_id):
    #return await self.subscribe({'type': 'timelineDetail', 'orderId': order_id})
    return await self.subscribe({'type': 'timelineDetailV2', 'orderId': order_id})

async def timeline_detail_savings_plan(self, savings_plan_id):
    #return await self.subscribe({'type': 'timelineDetail', 'savingsPlanId': savings_plan_id})
    return await self.subscribe({'type': 'timelineDetailV2', 'savingsPlanId': savings_plan_id})

changing the types.

# dl.py
if subscription['type'] == 'timelineTransactions':
    await self.tl.get_next_timeline(response, max_age_timestamp=self.since_timestamp)
elif subscription['type'] == 'timelineDetailV2':
    await self.tl.timelineDetail(response, self, max_age_timestamp=self.since_timestamp)
else:
    self.log.warning(f"unmatched subscription of type '{subscription['type']}':\n{preview(response)}")

And in the recieved message i got a valid document link as the payload: api DEBUG Received message: '1295 A ..

{
    "id": "2ac5zecb-ae38-424f-8a57-55b5cdba65b4",
    "sections": [
        {...},
        {...},
        {...},
        {...},
        {
            "title": "Dokumente",
            "data": [
                {
                    "title": "Abrechnung Ausführung",
                    "detail": "20.06.2024",
                    "action": {
                        "type": "browserModal",
                        "payload": "https://traderepublic-data-production.s3.eu-central-1.amazonaws.com/timeline/postbox/2024/6/20/..."
                    },
                    "id": "c524ee99-462e-4b9d-bf9c-6be2b932643f",
                    "postboxType": "SAVINGS_PLAN_EXECUTED_V2"
                }
            ],
            "action": null,
            "type": "documents"
        }
    ]
}

Maybe it helps someone with better python knowledge :D

thanks.

[moeff]

someone can fix this plz :D

[BjBe82]

Based on @tcdev0 i continued the reverse engineering of the new data structure returned by TR. As it looks like the structure has been changed in several places (i am not aware how the old structure looked like).

I adapted the code and tried to fix the title text and subtitle text as good as possible to generate reasonable pdf names.

There are patterns where i have no example (i.e. 'Wertpapierübertrag') and in cases where no title can be found an error 'Unable to detect titleText from response' is reported. In debug mode also the json will be printed which cannot be processed.

For now, I checked this with:

• "Dividende" old pattern -> (\Abrechnung) // new pattern -> (\Dokumente)
• "Sparplan" (\Sparplan\Abrechnung Ausführung)
• "Zinsen" (\Abrechnung)
• "Kauf"/"Verkauf" (\Abrechnung)

There are files which are not downloaded (not sure when this stopped working), for example files in:

• "Mistrade Regelungen"
• "Risikohinweise Crypto"
• "Steuerreport"
• .....

Created a pull request here: https://github.com/pytr-org/pytr/pull/73

TBD: The folder structure is based on the "documents/data/tile" which has for example changed for Dividends (old version vs new version). The question is if it would be better to enforce a fixed structure like:

• "Dividende"
• "Zinsen"
• "Kauf"/"Verkauf"
• "Others"

[BjBe82]

Solved for me with the new update to master. Just as a note, new dividend pdfs are in "\Dokumente" and no longer in "\Abrechnung".

[fft1010]

Hi alltogether, I think I messed my system up. At least I found two installed pytr.

% locate pytr | grep utils.py
/opt/homebrew/lib/python3.11/site-packages/pytr/utils.py
/usr/local/lib/python3.10/site-packages/pytr/utils.py

% which pytr                 
/opt/homebrew/bin/pytr

can a kind soul give me a tip, what to do, or "simply" deleting both path and re-installing it new? (I do not want to screw the system up more, than it is already.)

thanks juergen

[tjalb]

with python3.10 -m pip uninstall pytr you should be able to uninstall the second one, cause it seems that pytr points to the first one, cause it's in the homebrew environment.

[moeff]

thx for fixing