search

dhojayev / traderepublic-portfolio-downloader

App that downloads transactions and PDF documents from Trade Republic depot
GNU General Public License v3.0
14 stars 0 forks source link

cannot unmarshal object into Go struct field TransactionResponse.items.subAmount of type string #27

Closed dhojayev closed 3 months ago

dhojayev commented 3 months ago

Describe the bug Fetching timeline transactions fails on JSON unmarshalling stage because we never expected that subAmount field could containing something else but string.

Jun  5 21:27:33.866 [PANI] could not fetch transactions: could not unmarshal timelineTransactions response: json: cannot unmarshal object into Go struct field TransactionResponse.items.subAmount of type string
panic: (*logrus.Entry) 0xc0002b0a80

goroutine 1 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc0002b0150, 0x0, {0xc000338180, 0xb6})
        /go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x491
github.com/sirupsen/logrus.(*Entry).Log(0xc0002b0150, 0x0, {0xc0000a5e60?, 0xc0001d1180?, 0xc0002a4ab0?})
        /go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x48
github.com/sirupsen/logrus.(*Logger).Log(0xc0001d1180, 0x0, {0xc0000a5e60, 0x1, 0x1})
        /go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/logger.go:204 +0x58
github.com/sirupsen/logrus.(*Logger).Panic(...)
        /go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/logger.go:253
main.main()
        /github/workspace/cmd/portfoliodownloader/public/main.go:44 +0x43b

To Reproduce Waiting for the sample JSON

Expected behavior It should not fail on this stage, especially because we do not even use this field anywhere within our code.

tenjaa commented 3 months ago

Do you have a way to share the JSON privately? Or should I go through it and kill all personal data?

tenjaa commented 3 months ago 
    {
      "action": {
        "payload": "48efbcad-f320-5979-8d64-<XXX>",
        "type": "timelineDetail"
      },
      "amount": {
        "currency": "EUR",
        "fractionDigits": 2,
        "value": -5.95
      },
      "badge": null,
      "eventType": "card_successful_transaction",
      "icon": "logos/merchant-d86e2356-3f2c-4e12-bb84-<XXX>/v2",
      "id": "48efbcad-f320-5979-8d64-<XXX>",
      "status": "EXECUTED",
      "subAmount": null,
      "subtitle": null,
      "timestamp": "2024-05-27T13:51:55.167+0000",
      "title": "<XXX>"
    },
    {
      "action": {
        "payload": "8334ce43-36e0-43cc-9a23-<XXX>",
        "type": "timelineDetail"
      },
      "amount": {
        "currency": "EUR",
        "fractionDigits": 2,
        "value": -5.36
      },
      "badge": null,
      "eventType": "benefits_spare_change_execution",
      "icon": "logos/<XXX>/v2",
      "id": "8334ce43-36e0-43cc-9a23-<XXX>",
      "status": "EXECUTED",
      "subAmount": null,
      "subtitle": "Round up",
      "timestamp": "2024-05-23T13:59:05.402+0000",
      "title": "<XXX>"
    },
    {
      "action": {
        "payload": "701e379c-a992-520a-b5c9-<XXX>",
        "type": "timelineDetail"
      },
      "amount": {
        "currency": "EUR",
        "fractionDigits": 2,
        "value": -157.93
      },
      "badge": null,
      "eventType": "card_successful_transaction",
      "icon": "logos/merchant-fallback-entertainment/v2",
      "id": "701e379c-a992-520a-b5c9-<XXX>",
      "status": "EXECUTED",
      "subAmount": {
        "currency": "CZK",
        "fractionDigits": 2,
        "value": -3900
      },
      "subtitle": null,
      "timestamp": "2024-05-23T11:37:27.519+0000",
      "title": "<XXX>"
    },

I had a look myself and here are two examples of transactions with a subAmount that is not a string. Personal data and some parts of the ids are replaced by <XXX>.

dhojayev commented 3 months ago

Hey @tenjaa, thanks for the data. We could find a way to share the json privately, but I would still like all identifiable data to be removed. I can work with the examples you shared. I am already working on the fix for your case, it should be an easy hotfix for now. Will keep you posted.

dhojayev commented 3 months ago

Hey @tenjaa Please download the binary using the link below and check that it doesn't fail anymore. I fixed the issue and will release it if you could confirm that it works. https://github.com/dhojayev/traderepublic-portfolio-downloader/actions/runs/9402123472

Thanks

tenjaa commented 3 months ago

It starts downloading files successfully but then fails.

Jun  6 16:08:33.731 [INFO] [id:f8023427-d358-4263-9a18-17ff59a01ce7] Fetching transaction details (/home/runner/work/traderepublic-portfolio-downloader/traderepublic-portfolio-downloader/cmd/portfoliodownloader/app.go:59 github.com/dhojayev/traderepublic-portfolio-downloader/cmd/portfoliodownloader.App.Run)
Jun  6 16:08:33.731 [TRAC] [message:sub 74 {"id":"f8023427-d358-4263-9a18-17ff59a01ce7","token":"ey<XXX>","type":"timelineDetailV2"}] sent message (/home/runner/work/traderepublic-portfolio-downloader/traderepublic-portfolio-downloader/internal/api/websocket/reader.go:96 github.com/dhojayev/traderepublic-portfolio-downloader/internal/api/websocket.(*Reader).Read)
Jun  6 16:08:33.731 [TRAC] [message:73 C] received msg (/home/runner/work/traderepublic-portfolio-downloader/traderepublic-portfolio-downloader/internal/api/websocket/reader.go:104 github.com/dhojayev/traderepublic-portfolio-downloader/internal/api/websocket.(*Reader).Read)
Jun  6 16:08:33.759 [TRAC] [message:74 E {"errors":[{"errorCode":"NOT_FOUND","errorField":null,"errorMessage":"Could not find resource","meta":{"source":"PLATFORM"}}]}] received msg (/home/runner/work/traderepublic-portfolio-downloader/traderepublic-portfolio-downloader/internal/api/websocket/reader.go:104 github.com/dhojayev/traderepublic-portfolio-downloader/internal/api/websocket.(*Reader).Read)

Jun  6 16:06:56.718 [PANI] could not fetch transaction details: could not fetch timelineDetailV2: error state received: 75 E {"errors":[{"errorCode":"NOT_FOUND","errorField":null,"errorMessage":"Could not find resource","meta":{"source":"PLATFORM"}}]}
panic: (*logrus.Entry) 0xc00019ae00

goroutine 1 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc0003ac540, 0x0, {0xc0004f8380, 0xe0})
        /home/runner/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x491
github.com/sirupsen/logrus.(*Entry).Log(0xc0003ac540, 0x0, {0xc0004dfe60?, 0xc0002e4e30?, 0xc0002506f0?})
        /home/runner/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x48
github.com/sirupsen/logrus.(*Logger).Log(0xc0001d1180, 0x0, {0xc0004dfe60, 0x1, 0x1})
        /home/runner/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/logger.go:204 +0x58
github.com/sirupsen/logrus.(*Logger).Panic(...)
        /home/runner/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/logger.go:253
main.main()
        /home/runner/work/traderepublic-portfolio-downloader/traderepublic-portfolio-downloader/cmd/portfoliodownloader/public/main.go:44 +0x43b
dhojayev commented 3 months ago

This is interesting, it is definitely something else. could you post the json for this ID please? I will meanwhile disable exiting when details cannot be fetched but rather skip the transactions for now. But your json still could help to understand and fix the problem

tenjaa commented 3 months ago

I don't think I can. I enabled to write responses but there is no file called f8...json in responses/timelineDetailV2 . Or do I have to look for something else?

dhojayev commented 3 months ago

It should be in responses/timelineTransactions with the filename like page-X.json. Just try searching by content please. responses/timelineDetailV2 cannot yet contain anything as it is exactly the problem you facing, that the app cannot fetch the details, Trade Republic says that details for that ID do not exist.

tenjaa commented 3 months ago

Found it

    {
      "action": {
        "payload": "f8023427-d358-4263-9a18-17ff59a01ce7",
        "type": "timelineDetail"
      },
      "amount": {
        "currency": "EUR",
        "fractionDigits": 2,
        "value": 13.67
      },
      "badge": null,
      "eventType": "ORDER_EXECUTED",
      "icon": "logos/timeline_person/v2",
      "id": "f8023427-d358-4263-9a18-17ff59a01ce7",
      "status": "EXECUTED",
      "subAmount": null,
      "subtitle": "Verkaufsorder",
      "timestamp": "2021-09-22T12:06:16.619+0000",
      "title": "<XXX>"
    },
dhojayev commented 3 months ago

This is super strange. You can see in the action section it has an ID to fetch timelineDetail, which we are doing in this case, but then TR responds that the object doesn't exist. I guess that it happens because this order is old it requires the outdated websocket message type to fetch the details. But then I am not sure if the response there will be similar to timelineDetailV2. So I will simply skip it for now. But it was really helpful that you found it. Unfortunately, this means that this transaction won't be in your CSV, but we can work further together to see if implementing support for it is possible or not. Let me quickly finish skipping it for now and I will come back with the plan on what to do next.

dhojayev commented 3 months ago

@tenjaa If it would even be possible it would really help if you could find this transaction in TR mobile or web app (I know this means a lot of scrolling, and this is exactly the reason why our app exists :)) and check if you can press on it and see the details page. Thank you!

dhojayev commented 3 months ago

@tenjaa please download the new revision from here and this time I hope you will have your csv and docs :) https://github.com/dhojayev/traderepublic-portfolio-downloader/actions/runs/9403979524

tenjaa commented 3 months ago

@tenjaa If it would even be possible it would really help if you could find this transaction in TR mobile or web app (I know this means a lot of scrolling, and this is exactly the reason why our app exists :)) and check if you can press on it and see the details page. Thank you!

Doesn't work. I found it but when clicking on it it seems to load indefinately. There is no timeout but it is in that state for over a minute now already.

tenjaa commented 3 months ago

@tenjaa please download the new revision from here and this time I hope you will have your csv and docs :) https://github.com/dhojayev/traderepublic-portfolio-downloader/actions/runs/9403979524

It still shows an error but does not crash and continues successfully!

could not fetch transaction details: could not fetch timelineDetailV2: error state received: 75 E {"errors":[{"errorCode":"NOT_FOUND","errorField":null,"errorMessage":"Could not find resource","meta":{"source":"PLATFORM"}}]}
dhojayev commented 3 months ago

@tenjaa If it would even be possible it would really help if you could find this transaction in TR mobile or web app (I know this means a lot of scrolling, and this is exactly the reason why our app exists :)) and check if you can press on it and see the details page. Thank you!

Doesn't work. I found it but when clicking on it it seems to load indefinately. There is no timeout but it is in that state for over a minute now already.

Thanks a lot for spending time and confirming my theory. Even TR app cannot not fetch the info, so maybe we could use the data from the first response only, I will create an issue for that later.

dhojayev commented 3 months ago

@tenjaa please download the new revision from here and this time I hope you will have your csv and docs :) https://github.com/dhojayev/traderepublic-portfolio-downloader/actions/runs/9403979524

It still shows an error but does not crash and continues successfully!


could not fetch transaction details: could not fetch timelineDetailV2: error state received: 75 E {"errors":[{"errorCode":"NOT_FOUND","errorField":null,"errorMessage":"Could not find resource","meta":{"source":"PLATFORM"}}]}

This is exactly how I wanted it to be: error is shown to you, but the app doesn't fail. So I can release it if it finished all transactions with no errors?

tenjaa commented 3 months ago

In my opinion yes. There are still over 50% of documents skipped for me but I guess that is somewhat expected reading through the docs. In case something important might be missing I would just open another issue :)

dhojayev commented 3 months ago

Great. Which other documents you mean: account related documents (like Jahressteuerbericht, etc) or other transaction-related documents? I can prioritise implementation if there is request.

In my opinion yes. There are still over 50% of documents skipped for me but I guess that is somewhat expected reading through the docs. In case something important might be missing I would just open another issue :)

tenjaa commented 3 months ago

In the end my goal is to put all trades into portfolio performance.

There are two ways to import data: csv and parsing the pdf directly (see here). So my first thought was to download all pdfs and import them. I'll report back if something is missing or not working for my flow. Thanks!

dhojayev commented 3 months ago

In the end my goal is to put all trades into portfolio performance.

There are two ways to import data: csv and parsing the pdf directly (see here). So my first thought was to download all pdfs and import them. I'll report back if something is missing or not working for my flow. Thanks!

I will have a look into it. Was not familiar with that app. Maybe it would make sense to consider exporting csv for it in future. Thank you for your assistance.

martinAtGitHub commented 3 months ago

In my opinion yes. There are still over 50% of documents skipped for me but I guess that is somewhat expected reading through the docs. In case something important might be missing I would just open another issue :)

for me it is the credit card transactions that are skipped as unknown transactions

dhojayev commented 3 months ago

In my opinion yes. There are still over 50% of documents skipped for me but I guess that is somewhat expected reading through the docs. In case something important might be missing I would just open another issue :)

for me it is the credit card transactions that are skipped as unknown transactions

Hey @martinAtGitHub, Is it "unknown" or "unsupported"? If the latter, then this is by design as we only wanted investment related transactions to be in the csv file. There was an idea to create a separate csv file for card transactions, but it has not been planned properly till now. Plus, card transactions contain no documents attached to them, so there is not much data to extract from there.

martinAtGitHub commented 3 months ago

Hey @martinAtGitHub, Is it "unknown" or "unsupported"? If the latter, then this is by design as we only wanted investment related transactions to be in the csv file. There was an idea to create a separate csv file for card transactions, but it has not been planned properly till now. Plus, card transactions contain no documents attached to them, so there is not much data to extract from there.

Thanks for the response! It is "Unsupported". So as of now it is the intended behaviour. Since there is not that much reporting functionality in traderepublic it would be quite convinient, to have the credid card transactions in a CVS file.

dhojayev commented 3 months ago

Hey @martinAtGitHub, Is it "unknown" or "unsupported"? If the latter, then this is by design as we only wanted investment related transactions to be in the csv file. There was an idea to create a separate csv file for card transactions, but it has not been planned properly till now. Plus, card transactions contain no documents attached to them, so there is not much data to extract from there.

Thanks for the response! It is "Unsupported". So as of now it is the intended behaviour. Since there is not that much reporting functionality in traderepublic it would be quite convinient, to have the credid card transactions in a CVS file.

If that helps I can implement including those transactions optionally. But there are a few questions that would be good to clarify from the very beginning:

martinAtGitHub commented 3 months ago

If that helps I can implement including those transactions optionally. But there are a few questions that would be good to clarify from the very beginning:

* Shall we include those transactions in the same csv file?
* If yes - shall we simply write them under the "credit" and "debit" columns?

I would take the option that causes the least effort for you.

dhojayev commented 3 months ago

I created a dedicated issue for that. We can definitely implement it if it is helpful to at least one person. We already receive all necessary information from TR, so it is not that much of a hassle. However, I still don't know if writing those transactions into the same csv file is a good idea. I am thinking about having a separate file for that. I guess we all agree that it is not part of the portfolio but rather the "bank side" of Trade Republic.

martinAtGitHub commented 2 months ago

a separate file would be perfectly fine for me, too.