search

pytr-org / pytr

Use TradeRepublic in terminal and mass download all documents
https://pypi.org/project/pytr/
MIT License
425 stars 76 forks source link

event.shares has €-sign and causes an error decimal.InvalidOperation in transactions.py -> format_decimal(event.shares #104

Closed b166er closed 4 months ago

b166er commented 4 months ago

Description of the bug

Sometimes (at corporate bonds) a € sign appears next to a number in the event/details/sections/"Transaktionen"/data/detail/text. In transactions line 70 event.shares tries to format, the character causes an exception. the event.shartes are string-replaced from comma to dot in event.py line 69 to get a valid number. i guess you also have to remove the euro-sign to keep the number valid.

To Reproduce you have to buy a corporate bond, then you get the following subnode in the “events” in the API return:

"details": {"sections": [{"title": "Transaktion","data": [{"title": "*", "detail": {"text": "1,00 €",..."type": "text"

here is the full dataset of this event (anonymized)

{
    "id": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
    "timestamp": "2020-01-01T01:02:03.012+0000",
    "title": "Anleihe Januar 2020",
    "icon": "logos/XX0000123456/v2",
    "badge": null,
    "subtitle": "Verkaufsorder",
    "amount": {
      "currency": "EUR",
      "value": 1001.23,
      "fractionDigits": 2
    },
    "subAmount": null,
    "status": "EXECUTED",
    "action": {
      "type": "timelineDetail",
      "payload": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
    },
    "eventType": "ORDER_EXECUTED",
    "source": "timelineTransaction",
    "details": {
      "id": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
      "sections": [
        {
          "title": "Du hast 1.234,56 €  erhalten",
          "data": {
            "icon": "logos/XX0000123456/v2",
            "subtitleText": null,
            "timestamp": "2020-01-01T01:02:03.012+0000",
            "status": "executed"
          },
          "action": {
            "type": "instrumentDetail",
            "payload": "XS1234567890"
          },
          "type": "header"
        },
        {
          "title": "Übersicht",
          "data": [
            {
              "title": "Status",
              "detail": {
                "text": "Ausgeführt",
                "functionalStyle": "EXECUTED",
                "type": "status"
              },
              "style": "plain"
            },
            {
              "title": "Orderart",
              "detail": {
                "text": "Verkauf",
                "trend": null,
                "action": null,
                "type": "text"
              },
              "style": "plain"
            },
            {
              "title": "Asset",
              "detail": {
                "text": "Anleihe Januar 2020",
                "trend": null,
                "action": null,
                "type": "text"
              },
              "style": "plain"
            }
          ],
          "action": null,
          "type": "table"
        },
        {
          "title": "Performance",
          "data": [
            {
              "title": "Rendite",
              "detail": {
                "text": "0,12 %",
                "trend": "positive",
                "action": null,
                "type": "text"
              },
              "style": "plain"
            },
            {
              "title": "Gewinn",
              "detail": {
                "text": "1,23 €",
                "trend": "positive",
                "action": null,
                "type": "text"
              },
              "style": "plain"
            }
          ],
          "action": null,
          "type": "horizontalTable"
        },
        {
          "title": "Transaktion",
          "data": [
            {
              "title": "Gebühr",
              "detail": {
                "text": "1,00 €",
                "trend": null,
                "action": null,
                "type": "text"
              },
              "style": "plain"
            },
            {
              "title": "Gesamt",
              "detail": {
                "text": "+ 1.234,56 €",
                "trend": null,
                "action": null,
                "type": "text"
              },
              "style": "highlighted"
            }
          ],
          "action": {
            "type": "infoPage",
            "payload": {
              "sections": [
                {
                  "title": "Transaktion",
                  "action": null,
                  "type": "title"
                },
                {
                  "title": null,
                  "data": [
                    {
                      "title": "Nominale",
                      "detail": {
                        "text": "999,99 €",
                        "trend": null,
                        "action": null,
                        "type": "text"
                      },
                      "style": "plain"
                    },
                    {
                      "title": "Kurs",
                      "detail": {
                        "text": "99,99 %",
                        "trend": null,
                        "action": null,
                        "type": "text"
                      },
                      "style": "plain"
                    },
                    {
                      "title": "Order",
                      "detail": {
                        "text": "+ 999,99 €",
                        "trend": null,
                        "action": null,
                        "type": "text"
                      },
                      "style": "highlighted"
                    }
                  ],
                  "action": null,
                  "type": "table"
                },
                {
                  "title": null,
                  "data": [
                    {
                      "title": "Order",
                      "detail": {
                        "text": "+ 999,99 €",
                        "trend": null,
                        "action": null,
                        "type": "text"
                      },
                      "style": "plain"
                    },
                    {
                      "title": "Stückzinsen",
                      "detail": {
                        "text": "+ 12,34 €",
                        "trend": null,
                        "action": null,
                        "type": "text"
                      },
                      "style": "plain"
                    },
                    {
                      "title": "Gebühr",
                      "detail": {
                        "text": "1,00 €",
                        "trend": null,
                        "action": null,
                        "type": "text"
                      },
                      "style": "plain"
                    },
                    {
                      "title": "Gesamt",
                      "detail": {
                        "text": "+ 1.234,56 €",
                        "trend": null,
                        "action": null,
                        "type": "text"
                      },
                      "style": "highlighted"
                    }
                  ],
                  "action": null,
                  "type": "table"
                }
              ]
            }
          },
          "type": "table"
        },
        {
          "title": "Dokumente",
          "data": [
            {
              "title": "Abrechnung",
              "detail": "01.01.2020",
              "action": {
                "type": "browserModal",
                "payload": "https://blabla.cloud/timeline/postbox/2020/1/01/XXXX/pb12345678901234567890123456789.pdf?XXXX"
              },
              "id": "YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY",
              "postboxType": "SECURITIES_SETTLEMENT",
              "local filepath": "2020-01-01.01-02 Anleihe Abrechnung Januar 2020.pdf"
            },
            {
              "title": "Kosteninformation",
              "detail": "01.01.2020",
              "action": {
                "type": "browserModal",
                "payload": "https://blabla.cloud/timeline/postbox/2020/1/01/XXXX/pb12345678901234567890123456789.pdf?XXXX"
              },
              "id": "ZZZZZZZZ-ZZZZ-ZZZZ-ZZZZ-ZZZZZZZZZZZZ",
              "postboxType": "COSTS_INFO_SELL_V2",
              "local filepath": "2020-01-01.01-02 Anleihe Kosteninformation Januar 2020.pdf"
            }
          ],
          "action": null,
          "type": "documents"
        }
      ]
    },
    "has_docs": true
  },

Expected behavior no exception, all cases need to be replaced for clean number.

Error log

An Exception occurred: <class 'decimal.InvalidOperation'> - [<class 'decimal.ConversionSyntax'>]
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\dev\pytr\venv\Scripts\pytr.exe\__main__.py", line 7, in <module>
  File "C:\dev\pytr\venv\Lib\site-packages\pytr\main.py", line 208, in main
    asyncio.get_event_loop().run_until_complete(dl.dl_loop())
  File "C:\Python\3.12\Lib\asyncio\base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "C:\dev\pytr\venv\Lib\site-packages\pytr\dl.py", line 84, in dl_loop
    await self.tl.process_timelineDetail(response, self)
  File "C:\dev\pytr\venv\Lib\site-packages\pytr\timeline.py", line 164, in process_timelineDetail
    export_transactions(dl.output_path / 'all_events.json', dl.output_path / 'account_transactions.csv')
  File "C:\dev\pytr\venv\Lib\site-packages\pytr\transactions.py", line 72, in export_transactions
    shares = format_decimal(event.shares, locale=lang) if event.shares else ""
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\dev\pytr\venv\Lib\site-packages\babel\numbers.py", line 578, in format_decimal
    return pattern.apply(
           ^^^^^^^^^^^^^^
  File "C:\dev\pytr\venv\Lib\site-packages\babel\numbers.py", line 1402, in apply
    value = decimal.Decimal(str(value))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
decimal.InvalidOperation: [<class 'decimal.ConversionSyntax'>]

Proposed solution in event.py line 69

    def determine_shares(self):
        if self.pp_type == "TRADE_INVOICE":
            sections = self.event.get("details", {}).get("sections", [{}])
            for section in sections:
                if section.get("title") == "Transaktion":
                    self.shares = section.get("data", [{}])[0]["detail"][
                        "text"
                    ].replace(",", ".")
+                   # remove all non-numeric characters from string
+                   self.shares = re.sub('[^\.\d-]', '', self.shares)
+                   return self.shares # break from loop

    def determine_isin(self):
        if self.pp_type in ("DIVIDENDS", "TRADE_INVOICE"):
            sections = self.event.get("details", {}).get("sections", [{}])
            self.isin = self.event.get("icon", "")
            self.isin = self.isin[self.isin.find("/") + 1 :]
            self.isin = self.isin[: self.isin.find("/")]
            isin2 = self.isin
            for section in sections:
                action = section.get("action", None)
                if action and action.get("type", {}) == "instrumentDetail":
                    isin2 = section.get("action", {}).get("payload")
+                   break # break from loop
            if self.isin != isin2:
                self.isin = isin2

I'm not sure, maybe there are other currency signs that could be there.

Environment

melkor0 commented 4 months ago

Thanks for the issue report. I stumbled across the same error. I had to amend your proposed solution slightly: Either add

import re

or add

from re import sub

and subsequently modify

self.shares = re.sub('[^\.\d-]', '', self.shares)

to

self.shares = sub('[^\.\d-]', '', self.shares)

This is what I needed what I needed to get pytr running on my barebones windows Python 3.11.5 version.

Katzmann1983 commented 4 months ago

Fixed in #106