Open b166er opened 4 months ago
The idea is great. At the moment the folder structure is a mess, especially in case of non english accounts. My dl_docs has folders like this: Abrechnung Abrechnung Ausführung Abschlüsse Auftragsbestätigung Ausführungsanzeige Basisinformationsblatt Contrato con el cliente Cryptoauszug Dividende Ex-Post Kosteninformation Hauptversammlung Información de protección de datos Información sobre la garantía de depósitos Kapitalmaßnahmen Kosteninformation Kundenanschreiben Kundenvereinbarung Listado de precios y servicios Löschbestätigung Saveback Sparplan Steuerreport Términos y condiciones de la oferta acción gratis Your terms of service for instant trading volume Zinsen
We should be using folder and file names without space, everything in English, with a proper translation to German, Spanish, etc.
and that is the goal of my suggestion, with a config-map-file every user can decide for himself where which file goes into which folder and how the folders are named and the structure looks like. the config shown above is only an example of how it could look like, the folder names can definitely be specified, and every user can then decide for himself at the end what the folder should be called. I think the “uknown” block is also important in the config-map-file, so that API changes can be recognized immediately (by placing documents in the “unknown” folder) and, if necessary, users can adjust them themselves in the config.
I like the idea too and implemented a first draft based on @b166er yaml (with some small changes to the variables) here: https://github.com/BjBe82/pytr/tree/Issue-98-Customizable-destination-folder-structure
If someone is willing to review/test. There are some patterns in unknown still which need a pattern in the yaml file, but for now i use this for testing.
This introduces two new dependencies:
The script will copy the default template from "config/file_destination_config__template.yaml" to the user directory "
good work @BjBe82 in most cases your solution works great, by your solution, I could already find more documents that were missing in the config-map! i noticed a few small things.
in the pattern “info_anzeigen1” there is an event_subtitle: “Umtausch/Bezug”, the “/” is interpreted as a directory delimiter and thus a subdirectory is created for this document. The same as in the pattern “portfolio_abschluess” with event_title: "Q\d/\d+ Abschluss". My suggestion is to clean all node element names (event_type, event_title, event_subtitle, section_title, documenttitle) to get a valid file name. the following special characters must be removed or replaced with e.g. "" or "-"..., to get a proper file name (at least in Windows): \ / : * ? “ < > |
After a run, some documents were stored in the “unknown” folder, although the patterns are present. After a short troubleshooting, I found out that the string “SHAREBOOKING” with re.match() is equal to both “SHAREBOOKING” and “SHAREBOOKING_CANCELED”! i think we need to rethink the strategy how we compare the node names. there is definitely the requirement to match with regex, but not all strings are allowed to be matched with regex. i added “_regex”-postfix in the patterns for testing to get a clear distinction in the patterns which should be matched 1:1 with strings and which are allowed to be matched with regex.
I think it would be nice if we could distinguish whether documents have no pattern found or several patterns were found at the same time for an event, this makes it easier to search for the errors later. i have therefore introduced another rule block that is only used for multiple results.
As @Katzmann1983 has already correctly criticized, I have tried to rename all rule names and target folder names in English so that the config is uniformly readable by everyone. I think the users can then later customize their own folder names in their preferred language.
@todo what else occurs to me, are the json node names (e.g. event_subtitle or document_title) dependent on the selected language in the app from the user account or do they always come in German? if the nodes are also available in other languages, then we need a translation table.
unfortunately i am not allowed to upload files with yaml-extension to tickets in this github project, so i am adding the content of the config file directly to the ticket. here is the updated config:
destination:
# valid for all blocks without explicit filename
default:
filename: "{iso_date}.{iso_time} {event_title}" # {event_title} = Wertpapier-/ETF-/Produkt-Name
# if pattern not found, use this block
unknown:
path: "Unknown/{section_title}/"
filename: "{iso_date}.{iso_time} {event_type} - {event_subtitle} - {document_title} - {event_title}"
# if pattern found multiple times, use this block
multiple_match:
path: "MultipleMatchPattern/{section_title}/"
filename: "{iso_date}.{iso_time} {event_type} - {event_subtitle} - {document_title} - {event_title}"
# stocks
stock_order_settlement:
pattern: [
{ event_type: "ORDER_EXECUTED", event_subtitle: "Kauforder", section_title: "Dokumente", document_title: "Abrechnung" }, # mit limit gekauft
{ event_type: "ORDER_EXECUTED", event_subtitle: "Limit-Buy-Order", section_title: "Dokumente", document_title: "Abrechnung" }, # mit limit gekauft
{ event_type: "ORDER_EXECUTED", event_subtitle: "Stop-Sell-Order", section_title: "Dokumente", document_title: "Abrechnung" }, # mit limit verkauft
{ event_type: "TRADE_INVOICE", event_subtitle: "Limit-Buy-Order", section_title: "Dokumente", document_title: "Abrechnung" }, # mit limit gekauft
{ event_type: "ORDER_EXECUTED", event_subtitle: "Verkaufsorder", section_title: "Dokumente", document_title: "Abrechnung" }, # verkauft
{ event_type: "STOCK_PERK_REFUNDED", event_subtitle: "Eingelöst", section_title: "Dokumente", document_title: "Abrechnung" }, # Aktiengeschenk
{ event_type: "SHAREBOOKING", event_subtitle: "Verkauf", section_title: "Dokumente", document_title_regex: "Abrechnung \\d" }, # Kapitalmassnahme
]
path: "Stocks/Settlement/{iso_date_year}/"
stock_order_cost_report:
pattern: [
{ event_type: "ORDER_CREATED", event_subtitle: "Limit-Buy-Order erstellt", section_title: "Dokumente", document_title: "Kosteninformation" }, # limit erstellt
{ event_type: "ORDER_EXECUTED", event_subtitle: "Limit-Buy-Order", section_title: "Dokumente", document_title: "Kosteninformation" },
{ event_type: "ORDER_EXECUTED", event_subtitle: "Stop-Sell-Order", section_title: "Dokumente", document_title: "Kosteninformation" },
{ event_type: "ORDER_EXECUTED", event_subtitle: "Kauforder", section_title: "Dokumente", document_title: "Kosteninformation" }, # ohne limit gekauft
{ event_type: "ORDER_EXECUTED", event_subtitle: "Verkaufsorder", section_title: "Dokumente", document_title: "Kosteninformation" }, # verkauft
{ event_type: "TRADE_INVOICE", event_subtitle: "Limit-Buy-Order", section_title: "Dokumente", document_title: "Kosteninformation" },
{ event_type: "STOCK_PERK_REFUNDED", event_subtitle: "Eingelöst", section_title: "Dokumente", document_title: "Kosteninformation" }, # Aktiengeschenk
{ event_type: "STOCK_PERK_REFUNDED", event_subtitle: "Eingelöst", section_title: "Dokumente", document_title_regex: "Kosteninformation \\d" }, # Aktiengeschenk
{ event_type: "EX_POST_COST_REPORT", event_title: "Ex-Post Kosteninformation", section_title: "Dokumente", document_title_regex: "Ex-Post Kosteninformation \\d+" },
]
path: "Stocks/Cost report/{iso_date_year}/"
stock_order_created:
pattern: [
{ event_type: "TRADE_INVOICE", event_subtitle: "Limit-Buy-Order", section_title: "Dokumente", document_title: "Auftragsbestätigung" },
{ event_type: "ORDER_CREATED", event_subtitle: "Limit-Buy-Order erstellt", section_title: "Dokumente", document_title: "Auftragsbestätigung" },
{ event_type: "ORDER_EXECUTED", event_subtitle: "Limit-Buy-Order", section_title: "Dokumente", document_title: "Auftragsbestätigung" },
{ event_type: "ORDER_EXECUTED", event_subtitle: "Stop-Sell-Order", section_title: "Dokumente", document_title: "Auftragsbestätigung" },
]
path: "Stocks/Order created/{iso_date_year}/"
stock_order_canceled:
pattern: [
{ event_type: "ORDER_CANCELED", event_subtitle: "Limit-Buy-Order storniert", section_title: "Dokumente", document_title: "Löschbestätigung" },
{ event_type: "TRADE_CANCELED", event_subtitle: "Verkauf-Abrechnung storniert", section_title: "Dokumente", document_title_regex: "Abrechnung \\d" },
{ event_type: "SAVINGS_PLAN_CANCELED", event_subtitle: "Sparplan storniert", section_title: "Dokumente", document_title: "Stornierungsbestätigung" },
]
path: "Stocks/Order canceled/{iso_date_year}/"
# stocks Notice
stock_notice1:
pattern: [
{ event_type: "EXERCISE", event_subtitle: "Ausübung", section_title: "Dokumente", document_title: "Kundenanschreiben" },
{ event_type: "SHAREBOOKING", event_subtitle: "Split", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
{ event_type: "SHAREBOOKING", event_subtitle: "Fusion", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
{ event_type: "SHAREBOOKING", event_subtitle: "Verkauf", section_title: "Dokumente", document_title_regex: "Ausführungsanzeige \\d" },
{ event_type: "SHAREBOOKING", event_subtitle: "Ausbuchung", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
{ event_type: "SHAREBOOKING", event_subtitle: "ISIN Wechsel", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
{ event_type: "SHAREBOOKING", event_subtitle: "Entflechtung", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
{ event_type: "SHAREBOOKING", event_subtitle: "Reverse Split", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
{ event_type: "SHAREBOOKING", event_subtitle: "Titelumtausch", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
{ event_type: "SHAREBOOKING", event_subtitle: "Umtausch/Bezug", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
{ event_type: "SHAREBOOKING", event_subtitle: "Reorganisation", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
{ event_type: "SHAREBOOKING", event_subtitle: "Stockdividende", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
{ event_type: "SHAREBOOKING", event_subtitle: "Kapitalreduktion", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
{ event_type: "SHAREBOOKING", event_subtitle: "Titelgleichstellung", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
{ event_type: "SHAREBOOKING", event_subtitle: "Wertlose Ausbuchung", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
{ event_type: "SHAREBOOKING", event_subtitle: "Effektive Auslieferung", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
{ event_type: "SHAREBOOKING", event_subtitle: "Kapitalerhöhung gegen Bar", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
{ event_type: "SHAREBOOKING", event_subtitle: "Kapitalerhöhung aus Gesellschaftsmitteln", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
]
path: "Stocks/Notice/{iso_date_year}/"
filename: "{iso_date}.{iso_time} {event_subtitle} {event_title}"
# split for better readability
stock_notice2:
pattern: [
{ event_type: "CORPORATE_ACTION", event_subtitle: "Kapitalherabsetzung", section_title: "Dokumente", document_title: "Abrechnung" },
{ event_type: "SHAREBOOKING_CANCELED", event_subtitle: "Entflechtung storniert", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
{ event_type: "SHAREBOOKING_CANCELED", event_subtitle: "Stockdividende storniert", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
{ event_type: "SHAREBOOKING_CANCELED", event_subtitle: "Kapitalerhöhung gegen Bar storniert", section_title: "Dokumente", document_title: "Ausführungsanzeige" },
]
path: "Stocks/Notice/{iso_date_year}/"
filename: "{iso_date}.{iso_time} {event_subtitle} {event_title}"
# split for better readability
stock_notice3:
pattern: [
{ event_type: "SHAREBOOKING_TRANSACTIONAL", event_subtitle: "Fusion", section_title: "Dokumente", document_title: "Abrechnung" },
{ event_type: "SHAREBOOKING_TRANSACTIONAL", event_subtitle: "Reinvestierung", section_title: "Dokumente", document_title: "Abrechnung" },
{ event_type: "SHAREBOOKING_TRANSACTIONAL", event_subtitle: "Umtausch/Bezug", section_title: "Dokumente", document_title: "Abrechnung" },
{ event_type: "SHAREBOOKING_TRANSACTIONAL", event_subtitle: "Zwangsübernahme", section_title: "Dokumente", document_title: "Abrechnung" },
{ event_type: "SHAREBOOKING_TRANSACTIONAL", event_subtitle: "Kapitalreduktion", section_title: "Dokumente", document_title: "Abrechnung" },
# Capital increase in exchange for cash / Kapitalerhöhung gegen Bar
{ event_type: "INSTRUCTION_CORPORATE_ACTION", event_subtitle: "Weisung zur Aktiendividende", section_title: "Dokumente", document_title: "Kundenanschreiben" },
{ event_type: "INSTRUCTION_CORPORATE_ACTION", event_subtitle: "Weisung zur Aktiendividende", section_title: "Dokumente", document_title_regex: "Kundenanschreiben \\d" },
]
path: "Stocks/Notice/{iso_date_year}/"
filename: "{iso_date}.{iso_time} {event_subtitle} {event_title}"
# split for better readability
stock_notice4:
pattern: [
{ event_type: "ssp_corporate_action_invoice_shares", event_subtitle: "Spin-off", section_title: "Dokumente", document_title: "Dokumente" },
{ event_type: "ssp_corporate_action_invoice_shares", event_subtitle: "Reverse Split", section_title: "Dokumente", document_title: "Dokumente" },
# Recognition of subscription rights / Einbuchung Bezugsrechte
{ event_type: "ssp_corporate_action_invoice_shares", event_subtitle: "Zwischenvertrieb von Wertpapieren", section_title: "Dokumente", document_title: "Dokumente" },
{ event_type: "ssp_corporate_action_informative_notification", event_subtitle: "Wechsel", section_title: "Dokumente", document_title: "Information" },
{ event_type: "ssp_corporate_action_informative_notification", event_subtitle: "Information", section_title: "Dokumente", document_title: "Information" },
# Repurchase of interim securities / Rückkauf von Zwischentiteln
{ event_type: "ssp_corporate_action_informative_notification", event_subtitle: "Aufruf von Zwischenpapieren", section_title: "Dokumente", document_title: "Kapitalmaßnahmen" },
]
path: "Stocks/Notice/{iso_date_year}/"
filename: "{iso_date}.{iso_time} {event_subtitle} {event_title}"
stock_report:
pattern: [
{ event_type: "QUARTERLY_REPORT", event_title_regex: "Q\\d/\\d+ Abschluss", section_title: "Dokumente", document_title: "Kontoauszug" },
{ event_type: "QUARTERLY_REPORT", event_title_regex: "Q\\d/\\d+ Abschluss", section_title: "Dokumente", document_title: "Depotauszug" },
{ event_type: "QUARTERLY_REPORT", event_title_regex: "Q\\d/\\d+ Abschluss", section_title: "Dokumente", document_title: "Cryptoauszug" },
]
path: "Stocks/Report/{iso_date_year}/"
filename: "{iso_date} {document_title} {event_title}"
# General Meetings
stock_general_meetings:
pattern: [
{ event_type: "GENERAL_MEETING", event_subtitle: "Hauptversammlung", section_title: "Dokumente", document_title: "Hauptversammlung" },
{ event_type: "GENERAL_MEETING", event_subtitle: "Hauptversammlung", section_title: "Dokumente", document_title_regex: "Hauptversammlung \\d" },
{ event_type: "ssp_corporate_action_informative_notification", event_subtitle: "Jährliche Hauptversammlung", section_title: "Dokumente", document_title: "Information" },
{ event_type: "ssp_corporate_action_informative_notification", event_subtitle: "Außerordentliche oder spezielle Hauptversammlung", section_title: "Dokumente", document_title: "Information" },
]
path: "Stocks/General Meetings/{iso_date_year}/"
# Savings plan
savings_plan:
pattern: [
{ event_type: "SAVINGS_PLAN_INVOICE_CREATED", event_subtitle: "Sparplan ausgeführt", section_title: "Dokumente", document_title: "Abrechnung Ausführung" },
{ event_type: "SAVINGS_PLAN_EXECUTED", event_subtitle: "Sparplan ausgeführt", section_title: "Dokumente", document_title: "Abrechnung Ausführung" },
]
path: "Stocks/Savings plan/{iso_date_year}/"
# pre-determined tax base earning
stock_pre_earning_tax:
pattern: [
{ event_type: "PRE_DETERMINED_TAX_BASE_EARNING", event_subtitle: "Vorabpauschale", section_title: "Dokumente", document_title: "Vorabpauschale" },
]
path: "Stocks/PreEarningTax/{iso_date_year}/"
# Dividends
dividends_received:
pattern: [
{ event_type: "CREDIT", event_subtitle: "Dividende", section_title: "Dokumente", document_title: "Abrechnung" },
{ event_type: "CREDIT", event_subtitle: "Ausschüttung", section_title: "Dokumente", document_title: "Abrechnung" },
{ event_type: "ssp_corporate_action_invoice_cash", event_subtitle: "Bardividende", section_title: "Dokumente", document_title: "Dokumente" },
{ event_type: "ssp_corporate_action_invoice_cash", event_subtitle: "Dividende Wahlweise", section_title: "Dokumente", document_title: "Dokumente" },
{ event_type: "ssp_corporate_action_invoice_cash", event_subtitle: "Bardividende korrigiert", section_title: "Dokumente", document_title: "Dokumente" },
{ event_type: "ssp_corporate_action_invoice_cash", event_subtitle: "Reinvestition der Dividende", section_title: "Dokumente", document_title: "Dokumente" },
]
path: "Dividends/{iso_date_year}/"
# Dividends Corporate action election
dividends_election:
pattern: [
{ event_type: "INSTRUCTION_CORPORATE_ACTION", event_subtitle: "Weisung zur Aktiendividende", section_title: "Dokumente", document_title: "Dividende Wahlweise" },
{ event_type: "ssp_dividend_option_customer_instruction", event_subtitle: "Cash oder Aktie", section_title: "Dokumente", document_title: "Dividende Wahlweise" },
{ event_type: "ssp_corporate_action_informative_notification", event_subtitle: "Dividende Wahlweise", section_title: "Dokumente", document_title: "Kapitalmaßnahmen" },
]
path: "Dividendelection/{iso_date_year}/"
# bonds
bond_repayment:
pattern: [
{ event_type: "REPAYMENT", event_subtitle: "Tilgung", section_title: "Dokumente", document_title: "Abrechnung" },
]
path: "Bonds/{iso_date_year}/"
filename: "{iso_date}.{iso_time} Repayment {event_title}"
bond_interest:
pattern: [
{ event_type: "COUPON_PAYMENT", event_subtitle: "Coupon Zahlung", section_title: "Dokumente", document_title: "Abrechnung" },
]
path: "Bonds/{iso_date_year}/"
filename: "{iso_date}.{iso_time} Interest {event_title}"
# Saveback
saveback_enabled:
pattern: [
{ event_type: "benefits_saveback_execution", event_subtitle: "Saveback", section_title: "Dokumente", document_title: "Aktivierung" },
]
path: "Saveback/Enabled/"
saveback_executed:
pattern: [
{ event_type: "benefits_saveback_execution", event_subtitle: "Saveback", section_title: "Dokumente", document_title: "Abrechnung Ausführung" },
]
path: "Saveback/Report/"
saveback_cost_report:
pattern: [
{ event_type: "benefits_saveback_execution", event_subtitle: "Saveback", section_title: "Dokumente", document_title: "Kosteninformation" },
]
path: "Saveback/Cost report/"
# Round up
roundup_enabled:
pattern: [
# same files - multiple times at once
{ event_type: "benefits_spare_change_execution", event_subtitle: "Round up", section_title: "Dokumente", document_title: "Aktivierung" },
]
path: "Roundup/Enabled/"
roundup_executed:
pattern: [
{ event_type: "benefits_spare_change_execution", event_subtitle: "Round up", section_title: "Dokumente", document_title: "Abrechnung Ausführung" },
]
path: "Roundup/Report/{iso_date_year}/"
roundup_cost_report:
pattern: [
{ event_type: "benefits_spare_change_execution", event_subtitle: "Round up", section_title: "Dokumente", document_title: "Kosteninformation" },
]
path: "Roundup/Cost report/{iso_date_year}/"
# account
cash_interest:
pattern: [
{ event_type: "INTEREST_PAYOUT", event_title: "Zinsen", section_title: "Dokumente", document_title: "Abrechnung" },
{ event_type: "INTEREST_PAYOUT_CREATED", event_title: "Zinsen", section_title: "Dokumente", document_title: "Abrechnung" },
]
path: "Cash Interest/"
filename: "{iso_date} Zinsabrechnung"
cash_transfer_report:
pattern: [
{ event_type: "INCOMING_TRANSFER", event_subtitle: "Erhalten", section_title: "Dokument", document_title: "Transaktionsbestätigung" },
]
path: "Cash Report/"
filename: "{iso_date}.{iso_time} {document_title}" # {event_title} = Personal name
# annual tax report for account
account_tax_report:
pattern: [
{ event_type: "TAX_REFUND", event_subtitle: "Steuerbuchung", event_title: "Steuerabrechnung", section_title: "Dokumente", document_title: "Steuerabrechnung" },
{ event_type: "YEAR_END_TAX_REPORT", event_subtitle_regex: "Jahr \\d+", event_title: "Jährlicher Steuerreport", section_title: "Dokument", document_title_regex: "Steuerreport \\d+" },
]
path: "Tax/"
filename: "{iso_date} {document_title}"
account_tax_adjustment:
pattern: [
{ event_type: "ssp_tax_correction_invoice", event_title: "Steuerkorrektur", section_title: "Dokumente", document_title: "Steuerabrechnung" },
]
path: "Tax/"
filename: "{iso_date} {event_title}"
# common informations
notice_stocks:
pattern: [
{ event_type: "CUSTOMER_CREATED", event_subtitle: "Erhalten", section_title: "Dokumente", document_title: "Basisinformationen über Wertpapiere" },
{ event_type: "DOCUMENTS_CHANGED", event_subtitle: "Geändert", section_title: "Dokumente", document_title: "Kundenvereinbarung" }, # same file as below
{ event_type: "DOCUMENTS_CHANGED", event_subtitle: "Geändert", section_title: "Aktualisierte Dokumente", document_title: "Kundenvereinbarung" }, # same file as above
{ event_type: "DOCUMENTS_CHANGED", event_subtitle: "Geändert", section_title: "Dokumente", document_title: "Datenschutzinformationen" },
{ event_type: "DOCUMENTS_CHANGED", event_subtitle: "Geändert", section_title: "Aktualisierte Dokumente", document_title: "Datenschutzinformationen" },
{ event_type: "DOCUMENTS_CHANGED", event_subtitle: "Geändert", section_title: "Dokumente", document_title: "Info zur Einlagensicherung" },
{ event_type: "DOCUMENTS_CHANGED", event_subtitle: "Geändert", section_title: "Aktualisierte Dokumente", document_title: "Info zur Einlagensicherung" },
{ event_type: "DOCUMENTS_CHANGED", event_subtitle: "Geändert", section_title: "Dokumente", document_title: "Hinweise zu den Handelsplätzen" },
{ event_type: "DOCUMENTS_CHANGED", event_subtitle: "Geändert", section_title: "Aktualisierte Dokumente", document_title: "Hinweise zu den Handelsplätzen" },
{ event_type: "DOCUMENTS_CHANGED", event_subtitle: "Geändert", section_title: "Dokumente", document_title: "Preis- und Leistungsverzeichnis" },
{ event_type: "DOCUMENTS_CHANGED", event_subtitle: "Geändert", section_title: "Aktualisierte Dokumente", document_title: "Preis- und Leistungsverzeichnis" },
]
path: "Notice/"
filename: "{iso_date} {document_title}"
notice_crypto:
pattern: [
{ event_type: "DOCUMENTS_ACCEPTED", event_subtitle: "Angenommen", section_title: "Dokumente", document_title: "Kundenvereinbarung" },
{ event_type: "DOCUMENTS_ACCEPTED", event_subtitle: "Angenommen", section_title: "Dokumente", document_title: "Customer Agreement" },
{ event_type: "DOCUMENTS_ACCEPTED", event_subtitle: "Angenommen", section_title: "Dokumente", document_title: "Mistrade Regelungen" },
{ event_type: "DOCUMENTS_ACCEPTED", event_subtitle: "Angenommen", section_title: "Dokumente", document_title: "Risikohinweise Crypto" },
{ event_type: "DOCUMENTS_ACCEPTED", event_subtitle: "Angenommen", section_title: "Dokumente", document_title: "Crypto Verwahrbedingungen" },
{ event_type: "DOCUMENTS_ACCEPTED", event_subtitle: "Angenommen", section_title: "Dokumente", document_title: "Vorvertragliche Informationen" },
{ event_type: "DOCUMENTS_ACCEPTED", event_subtitle: "Angenommen", section_title: "Dokumente", document_title: "Widerrufsbelehrung Cryptoverwahrer" },
{ event_type: "DOCUMENTS_ACCEPTED", event_subtitle: "Angenommen", section_title: "Dokumente", document_title: "Datenschutzinformationen Cryptoverwahrer" },
]
path: "Notice/"
filename: "{iso_date} {document_title}"
notice_warrant_and_etfs:
pattern: [
# when you buy a product for the first time, you will receive information documents about this product
{ event_type: "ORDER_EXECUTED", event_subtitle: "Kauforder", section_title: "Dokumente", document_title: "Basisinformationsblatt" },
{ event_type: "GESH_CORPORATE_ACTION", event_subtitle: "Unternehmensmeldung", section_title: "Dokumente", document_title: "Kundenanschreiben" },
{ event_type: "GESH_CORPORATE_ACTION_MULTIPLE_POSITIONS", event_subtitle: "Gesellschaftshinweis", section_title: "Dokumente", document_title_regex: "Kundenanschreiben \\d" },
]
path: "Notice/"
filename: "{iso_date} {document_title} - {event_title}"
card_ordered:
pattern: [
{ event_type: "card_order_billed", event_title: "Trade Republic Card", section_title: "Dokumente", document_title: "Bestellung Trade Republic Karte" },
]
path: "Notice/"
filename: "{iso_date} {document_title}"
# The following events do not contain any documents, these nodes are skipped.
#
# payment_inbound:
# pattern: [
# { event_type: "PAYMENT_INBOUND", event_subtitle: , section_title: , document_title: },
# { event_type: "INCOMING_TRANSFER", event_subtitle: "Erhalten", section_title: , document_title: },
# ]
# path: "Cash Report/"
#
# payment_outbout:
# pattern: [
# { event_type: "PAYMENT_OUTBOUND", event_subtitle: , section_title: , document_title: },
# ]
# path: "Cash Report/"
#
# card_refund:
# pattern: [
# { event_type: "card_refund", event_subtitle: , section_title: , document_title: },
# ]
# path: "Cash Report/"
#
# card_successful_transaction:
# pattern: [
# { event_type: "card_successful_transaction", event_subtitle: , section_title: , document_title: },
# ]
# path: "Cash Report/"
#
# card_successful_atm_withdrawal:
# pattern: [
# { event_type: "card_successful_atm_withdrawal", event_subtitle: , event_title: "Abhebung", section_title: , document_title: },
# ]
# path: "Cash Report/"
#
#
# card_failed_transaction:
# pattern: [
# { event_type: "card_failed_transaction", event_subtitle: "Abgebrochen", section_title: , document_title: },
# ]
# path: "Cash Report/"
#
# card_successful_verification:
# pattern: [
# { event_type: "card_successful_verification", event_subtitle: , section_title: , document_title: },
# ]
# path: "Cash Report/"
#
# new_account_iban:
# pattern: [
# { event_type: "new_tr_iban", event_subtitle: , section_title: , document_title: },
# ]
# path: "Cash Report/"
#
# stock_order_rejected:
# pattern: [
# { event_type: "ORDER_REJECTED", event_subtitle: "Kauforder abgelehnt", section_title: , document_title: },
# ]
# path: "Stocks/Order canceled/{iso_date_year}/"
#
#
# stock_order_expired:
# pattern: [
# { event_type: "ORDER_EXPIRED", event_subtitle: "Kauforder abgelaufen", section_title: , document_title: },
# ]
# path: "Stocks/Order canceled/{iso_date_year}/"
#
Here is my suggestion for improving the points mentioned above.
diff --git a/pytr/file_destination_provider.py b/pytr/file_destination_provider.py
index 10e5287..e88249c 100644
--- a/pytr/file_destination_provider.py
+++ b/pytr/file_destination_provider.py
@@ -16,8 +16,9 @@ except ImportError:
from yaml import Loader, Dumper
-ALL_CONFIG = "all"
+ALL_CONFIG = "default"
UNKNOWN_CONFIG = "unknown"
+MULTIPLEMATCH_CONFIG = "multiple_match"
TEMPLATE_FILE_NAME ="file_destination_config__template.yaml"
@@ -36,12 +37,17 @@ class DestinationConfig:
class Pattern:
- def __init__(self, event_type: str, event_subtitle: str, event_title: str, section_title: str, document_title: str):
+ def __init__(self, event_type: str, event_subtitle: str, event_title: str, section_title: str, document_title: str, event_type_regex: str, event_subtitle_regex: str, event_title_regex: str, section_title_regex: str, document_title_regex: str):
self.event_type = event_type
self.event_subtitle = event_subtitle
self.event_title = event_title
self.section_title = section_title
self.document_title = document_title
+ self.event_type_regex = event_type_regex
+ self.event_subtitle_regex = event_subtitle_regex
+ self.event_title_regex = event_title_regex
+ self.section_title_regex = section_title_regex
+ self.document_title_regex = document_title_regex
class FileDestinationProvider:
@@ -51,7 +57,7 @@ class FileDestinationProvider:
A provider for file path and file names based on the event type and other parameters.
'''
self._log = get_logger(__name__)
-
+
config_file_path = Path(DESTINATION_CONFIG_FILE)
if config_file_path.is_file() == False:
self.__create_default_config(config_file_path)
@@ -62,7 +68,7 @@ class FileDestinationProvider:
self.__validate_config(destination_config)
destinations = destination_config["destination"]
-
+
self._destination_configs: list[DestinationConfig] = []
for config_name in destinations:
@@ -72,6 +78,9 @@ class FileDestinationProvider:
elif config_name == UNKNOWN_CONFIG:
self._unknown_file_config = DestinationConfig(
UNKNOWN_CONFIG, destinations[UNKNOWN_CONFIG]["filename"], destinations[UNKNOWN_CONFIG]["path"])
+ elif config_name == MULTIPLEMATCH_CONFIG:
+ self._multiplematch_file_config = DestinationConfig(
+ MULTIPLEMATCH_CONFIG, destinations[MULTIPLEMATCH_CONFIG]["filename"], destinations[MULTIPLEMATCH_CONFIG]["path"])
else:
patterns = self.__extract_pattern(
destinations[config_name].get("pattern", None))
@@ -91,6 +100,20 @@ class FileDestinationProvider:
variables (dict): The variables->value dict to be used in the file path and file name format.
'''
+ # invalid characters dict,
+ # for cleaning up the node names
+ # in order to be able to use them as valid filenames.
+ replacements_dict = {
+ '"': '',
+ '?': '',
+ '<': '',
+ '>': '',
+ '*': '',
+ '|': '-',
+ '/': '-',
+ '\\': '-'
+ }
+
matching_configs = self._destination_configs.copy()
# Maybe this can be improved looks like a lot of code duplication ... on the other hand using a
@@ -98,27 +121,27 @@ class FileDestinationProvider:
if event_type is not None:
matching_configs = list(filter(lambda config: self.__is_matching_config(
config, "event_type", event_type), matching_configs))
- variables["event_type"] = event_type
+ variables["event_type"] = event_type.translate(str.maketrans(replacements_dict)).strip()
if event_title is not None:
matching_configs = list(filter(lambda config: self.__is_matching_config(
config, "event_title", event_title), matching_configs))
- variables["event_title"] = event_title
+ variables["event_title"] = event_title.translate(str.maketrans(replacements_dict)).strip()
if event_subtitle is not None:
matching_configs = list(filter(lambda config: self.__is_matching_config(
config, "event_subtitle", event_subtitle), matching_configs))
- variables["event_subtitle"] = event_subtitle
+ variables["event_subtitle"] = event_subtitle.translate(str.maketrans(replacements_dict)).strip()
if section_title is not None:
matching_configs = list(filter(lambda config: self.__is_matching_config(
config, "section_title", section_title), matching_configs))
- variables["section_title"] = section_title
+ variables["section_title"] = section_title.translate(str.maketrans(replacements_dict)).strip()
if document_title is not None:
matching_configs = list(filter(lambda config: self.__is_matching_config(
config, "document_title", document_title), matching_configs))
- variables["document_title"] = document_title
+ variables["document_title"] = document_title.translate(str.maketrans(replacements_dict)).strip()
if len(matching_configs) == 0:
self._log.debug(
@@ -126,15 +149,21 @@ class FileDestinationProvider:
return self.__create_file_path(self._unknown_file_config, variables)
if len(matching_configs) > 1:
- self._log.debug(f"Multiple Destination Patterns where found. Using 'unknown' config! Parameter: event_type:{event_type}, event_title:{event_title},event_subtitle:{event_subtitle},section_title:{section_title},document_title:{document_title}")
- return self.__create_file_path(self._unknown_file_config, variables)
+ self._log.debug(f"Multiple Destination Patterns where found. Using 'multiple_match' config! Parameter: event_type:{event_type}, event_title:{event_title},event_subtitle:{event_subtitle},section_title:{section_title},document_title:{document_title}")
+ return self.__create_file_path(self._multiplematch_file_config, variables)
return self.__create_file_path(matching_configs[0], variables)
def __is_matching_config(self, config: DestinationConfig, key: str, value: str):
+ key_regex = key + '_regex'
for pattern in config.pattern:
attribute = getattr(pattern, key)
- if attribute is None or re.match(attribute, value):
+ attribute_regex = getattr(pattern, key_regex)
+ if attribute is None and attribute_regex is None:
+ return True
+ if attribute and attribute == value:
+ return True
+ if attribute_regex and re.match(attribute_regex, value):
return True
return False
@@ -156,7 +187,12 @@ class FileDestinationProvider:
pattern.get("event_subtitle", None),
pattern.get("event_title", None),
pattern.get("section_title", None),
- pattern.get("document_title", None)))
+ pattern.get("document_title", None),
+ pattern.get("event_type_regex", None),
+ pattern.get("event_subtitle_regex", None),
+ pattern.get("event_title_regex", None),
+ pattern.get("section_title_regex", None),
+ pattern.get("document_title_regex", None)))
return patterns
I hope this helps, my tests were all successful.
Thanks for the feedback so far. I am a little bit limited in time during the week but will take a look at it over the weekend.
I tied to address all the issues mentioned so far except one thing, the “_regex”-postfix. I am not sure if i like it since from a user perspective It's easy to miss and also in code i am not as happy. Furthermore, I think the main issue is that we need to use "fullmatch" for the regex, and basically if regex is used it must match the full string. I changed this in the current version.
I totally agree with you, your new code change works perfectly for me! My “unknown” and “MultipleMatch” folders are empty, I'm happy. I would love to see your pull-request approved into the main-branch.
Do we need the regex at all? I think the code would be easier to maintain without it. When playing with it myself, I just removed all these criteria and the outcome was the same
if i remove the nodes with the regex expressions, then the rule blocks become ambiguous because a missing node outputs a true https://github.com/BjBe82/pytr/blob/Issue-98-Customizable-destination-folder-structure/pytr/file_destination_provider.py#L149 and therefore many documents are stored in the MultiMatch folder.
Example: An event with the following values: event_type: “TRADE_INVOICE”, event_subtitle: “Limit buy order”, section_title: “Documents”, document_title: “Settlement”
matching_configs = list(filter(lambda config: self.__is_matching_config(config, “event_type”, event_type), matching_configs))
iterates each block in the config, filters blocks, if a pattern was found in the block, in this example the first and the second block marked as 1 & 2 in red on the screenshot. for each block there is a sub-loop, for each pattern line, to find a specific node (e.g. event_type) in https://github.com/BjBe82/pytr/blob/Issue-98-Customizable-destination-folder-structure/pytr/file_destination_provider.py#L146
def __is_matching_config(self, config: DestinationConfig, key: str, value: str):
for pattern in config.pattern:
attribute = getattr(pattern, key)
# only checks whether the value of “event_type” matches the input,
# but not whether the entire pattern line matches all nodes.
if attribute is None or re.fullmatch(attribute, value):
return True
return False
each node is currently checked individually and not in connection with other nodes per pattern line, marked as blue numbers in the screenshot. with green square for match / return true. It can therefore happen that a pattern line and therefore a rule block is found although the individual nodes are distributed across the block and are not in a single pattern line. I think this point needs to be reworked so that the nodes are all compared together per pattern row.
my suggestion, we could remove pattern rows where the nodes do not match to avoid checking in the next node loop with pattern rows that already do not match.
diff --git a/pytr/file_destination_provider.py b/pytr/file_destination_provider.py
index 10e5287..e88249c 100644
--- a/pytr/file_destination_provider.py
+++ b/pytr/file_destination_provider.py
@@ -145,1 +145,2 @@ def __is_matching_config(self, config: DestinationConfig, key: str, value: str):
- for pattern in config.pattern:
+ while len(config.pattern) > 0: # loop until patterns exists
+ pattern = config.pattern[0] # get first pattern line
attribute = getattr(pattern, key)
if attribute is None or re.fullmatch(attribute, value):
return True
+ else: # if value not match
+ config.pattern.pop(0) # remove pattern line from the block to avoid checking for next value
return False
I also think that the regex option would be a powerful function for selecting dynamic titles in the future. regex comparison doesn't actually take much performance. I am aware that not everyone can use regex. I think users without regex experience will continue to make requests via github-issues to find new documents with the rule config.
I agree with @b166er that regex can be a powerful feature to have finer grained pattern matching. From an end user perspective who just wants to download the docs, I think he expects a yaml file which just works out of the box without ever touching it. Maybe we can provide multiple languages as examples (but this is nice to have and plain translation work).
Also, we will not get this to work without any regex, for example i have this: The only parameter where this can be distinguished is the "document_title" which can be "Abrechnung", "Kosteninformation" and "Kosteninformation" appended with or without a number based on the fact how mayn transaction you have at the same day.
@b166er Ok i see the issue, but we cannot use config.pattern.pop(0)
. This will modify the loaded configuration and will cause issues for following calls. For example:
provider = FileDestinationProvider()
print(provider.get_file_path("TRADE_INVOICE", None, "Limit-Buy-Order", "Dokumente", "Kosteninformation", {"variable1": "value1"}))
print(provider.get_file_path("STOCK_PERK_REFUNDED", None, "Eingelöst", "Dokumente", "Kosteninformation", {"variable1": "value1"}))
print(provider.get_file_path("TRADE_INVOICE", None, "Limit-Buy-Order", "Dokumente", "Kosteninformation", {"variable1": "value1"}))
print(provider.get_file_path("STOCK_PERK_REFUNDED", None, "Eingelöst", "Dokumente", "Kosteninformation", {"variable1": "value1"}))
will print:
Stocks/Cost report/{iso_date_year}/{iso_date}.{iso_time} {event_title}
Stocks/Cost report/{iso_date_year}/{iso_date}.{iso_time} {event_title}
Unknown/Dokumente/{iso_date}.{iso_time} TRADE_INVOICE - Limit-Buy-Order - Kosteninformation - {event_title}
Unknown/Dokumente/{iso_date}.{iso_time} STOCK_PERK_REFUNDED - Eingelöst - Kosteninformation - {event_title}
We would need a deep copy of the config on each call to avoid this. Since i don't want to deeply copy this for each call, i changed the matching to check all pattern values and only accept full matches. The change is now in git.
Other changes: Patterns which are not found/need to be added or changed:
{ event_type: "ORDER_EXECUTED", event_subtitle: "Limit-Buy-Order", section_title: "Dokumente", document_title: "Basisinformationsblatt" },
{ event_type: "ORDER_CANCELED", event_subtitle: "Stop-Sell-Order storniert", section_title: "Dokumente", document_title: "Löschbestätigung" },
{ event_type: "CREDIT_CANCELED", event_subtitle: "Dividende", section_title: "Dokumente", document_title: "Abrechnung \\d" }
{ event_type: "TAX_CORRECTION", event_subtitle: "Steuerbuchung", event_title: "Steuerabrechnung", section_title: "Dokumente", document_title: "Abrechnung" },
{ event_type: "TRADE_INVOICE", event_subtitle: "Verkaufsorder", section_title: "Dokumente", document_title: "Abrechnung" },
{ event_type: "TRADE_INVOICE", event_subtitle: "Verkaufsorder", section_title: "Dokumente", document_title: "Kosteninformation" },
{ event_type: "YEAR_END_TAX_REPORT", event_subtitle: "Jahr \\d+", event_title: "Jährlicher Steuerreport", section_title: "Dokumente", document_title: "Steuerreport \\d+" }
{ event_type: "ORDER_EXECUTED", event_subtitle: "Kauforder", section_title: "Dokumente", document_title: "Abrechnung(.\d)?" },
This happens if you trade on the same day twice -> maybe pattern for other documents too.{ event_type: "ORDER_EXECUTED", event_subtitle: "Kauforder", section_title: "Dokumente", document_title: "Basisinformationsblatt(.\\d)?" },
{ event_type: "ORDER_EXECUTED", event_subtitle: "Kauforder", section_title: "Dokumente", document_title: "Kosteninformation(.\\d)?" }
To be discussed points:
(1) A pattern which can be observed is that if we have multiple trades per day, we will have documents with "document_title" that have a number appended. Maybe we should in general add (.\\d+)
to all "document_title" entries. This would also resolve entries like this
{ event_type: "STOCK_PERK_REFUNDED", event_subtitle: "Eingelöst", section_title: "Dokumente", document_title: "Kosteninformation" }, # Aktiengeschenk
{ event_type: "STOCK_PERK_REFUNDED", event_subtitle: "Eingelöst", section_title: "Dokumente", document_title: "Kosteninformation \\d" }, # Aktiengeschenk
to a single entry.
(2) Another thing what i was thinking about is that we might need to add a version number to the yaml file if we want to detect deprecated yaml configurations. For example, if TR changes the texts of the event, event title, subtitle and co we might need to adapt the config. And this change should be reflected somehow and reported to the user which might still use an older version.
I think it would be best to remove some details and make both the parser and the yaml file shorter and easier to maintain. Please have a look at this PR https://github.com/BjBe82/pytr/pull/2/files
Fixed the conflicts and accepted your pull request with the following changes:
i have tested the code from the pull-request branch, works perfectly! well done. I would be very happy to find this change in the release as soon as possible. here are my configuration changes to avoid triggering the MultiMatch and Unknown rule: file_destination_config__template.yaml.patch
Applied @b166er patch + appended all document titles with (.\\d+)?
just to be safe if there are multiple occurrences of the same file. There might be cases where this never happens, but just to have a consistent pattern, I also applied it to them since there is no downside to it.
Here is my current configuration version.
destination:
################################################################################################
## Default and fallback patterns used for the destination of a downloaded document.
################################################################################################
# valid for all blocks without explicit filename
default:
filename: "{iso_date}.{iso_time} {event_title}" # {event_title} = Wertpapier-/ETF-/Produkt-Name
# if pattern not found, use this block
unknown:
path: "Unknown/{section_title}/"
filename: "{iso_date}.{iso_time} {event_type} - {event_subtitle} - {document_title} - {event_title}"
# if pattern found multiple times, use this block
multiple_match:
path: "MultipleMatch/{section_title}/"
filename: "{iso_date}.{iso_time} {event_type} - {event_subtitle} - {document_title} - {event_title}"
################################################################################################
################################################################################################
## Specific patterns for the destination of a downloaded document.
################################################################################################
# stocks
stock_order_settlement:
pattern: [
{event_type: "ORDER_EXECUTED", document_title: "Abrechnung(.\\d+)?"}, # mit limit verkauft
{event_type: "TRADE_INVOICE", document_title: "Abrechnung(.\\d+)?"}, # mit limit gekauft
{event_type: "STOCK_PERK_REFUNDED", document_title: "Abrechnung(.\\d+)?"}, # Aktiengeschenk
{event_type: "SHAREBOOKING", document_title: "Abrechnung(.\\d+)?"}, # Kapitalmassnahme
]
path: "Stocks/Settlement/{iso_date_year}/"
stock_order_cost_report:
pattern: [
{event_type: "ORDER_CREATED", document_title: "Kosteninformation(.\\d+)?"}, # limit erstellt
{event_type: "ORDER_EXECUTED", document_title: "Kosteninformation(.\\d+)?"},
{event_type: "TRADE_INVOICE", document_title: "Kosteninformation(.\\d+)?"},
{event_type: "STOCK_PERK_REFUNDED", document_title: "Kosteninformation(.\\d+)?"}, # Aktiengeschenk
{event_type: "EX_POST_COST_REPORT"},
]
path: "Stocks/Cost report/{iso_date_year}/"
stock_order_created:
pattern: [
{event_type: "ORDER_CREATED", document_title: "Auftragsbestätigung(.\\d+)?"},
{event_type: "ORDER_EXECUTED", document_title: "Auftragsbestätigung(.\\d+)?"},
{event_type: "TRADE_INVOICE", document_title: "Auftragsbestätigung(.\\d+)?"},
]
path: "Stocks/Order created/{iso_date_year}/"
stock_order_canceled:
pattern: [{event_type: "TRADE_CANCELED"},{event_type: "ORDER_CANCELED"},]
path: "Stocks/Order canceled/{iso_date_year}/"
order_expired:
pattern: [{event_type: "ORDER_EXPIRED"}]
path: "Stocks/Order canceled/{iso_date_year}/"
filename: "{iso_date} {document_title}"
# Kapitalmaßnahmen
stock_notice_1:
pattern: [{event_type: "EXERCISE"}, {event_type: "SHAREBOOKING", document_title: "Ausführungsanzeige(.\\d+)?"}]
path: "Stocks/Notice/{iso_date_year}/"
filename: "{iso_date}.{iso_time} {event_subtitle} {event_title}"
# split for better readability
stock_notice_2:
pattern: [{event_type: "CORPORATE_ACTION"}, {event_type: "SHAREBOOKING_CANCELED"}]
path: "Stocks/Notice/{iso_date_year}/"
filename: "{iso_date}.{iso_time} {event_subtitle} {event_title}"
# split for better readability
stock_notice_3:
pattern: [
{event_type: "SHAREBOOKING_TRANSACTIONAL"},
{event_type: "INSTRUCTION_CORPORATE_ACTION", document_title: "Kundenanschreiben(.\\d+)?"}, #Kapitalerhöhung gegen Bar
]
path: "Stocks/Notice/{iso_date_year}/"
filename: "{iso_date}.{iso_time} {event_subtitle} {event_title}"
# split for better readability
stock_notice_4:
pattern: [
{event_type: "ssp_corporate_action_invoice_shares", event_subtitle: "Spin-off"},
{event_type: "ssp_corporate_action_invoice_shares", event_subtitle: "Reverse Split"},
{event_type: "ssp_corporate_action_invoice_shares", event_subtitle: "Aktiendividende"},
{event_type: "ssp_corporate_action_invoice_shares", event_subtitle: "Zwischenvertrieb von Wertpapieren"},
{event_type: "ssp_corporate_action_informative_notification", event_subtitle: "Wechsel"},
{event_type: "ssp_corporate_action_informative_notification", event_subtitle: "Information"},
{event_type: "ssp_corporate_action_informative_notification", event_subtitle: "Aufruf von Zwischenpapieren"},
]
path: "Stocks/Notice/{iso_date_year}/"
filename: "{iso_date}.{iso_time} {event_subtitle} - {event_title}"
stock_report:
pattern: [
{event_type: "QUARTERLY_REPORT", document_title: "Kontoauszug(.\\d+)?"},
{event_type: "QUARTERLY_REPORT", document_title: "Depotauszug(.\\d+)?"},
{event_type: "QUARTERLY_REPORT", document_title: "Cryptoauszug(.\\d+)?"},
]
path: "Stocks/Report/{iso_date_year}/"
filename: "{iso_date} {document_title} {event_title}"
# General Meetings
stock_general_meetings:
pattern: [
{event_type: "GENERAL_MEETING", document_title: "Hauptversammlung"},
{event_type: "ssp_corporate_action_informative_notification", event_subtitle: "Jährliche Hauptversammlung"},
]
path: "Stocks/General Meetings/{iso_date_year}/"
stock_general_meetings_multiple_files:
pattern: [{event_type: "GENERAL_MEETING", document_title: "Hauptversammlung \\d+"}]
path: "Stocks/General Meetings/{iso_date_year}/"
filename: "{iso_date}.{iso_time} {event_title} - {document_title}"
stock_special_meetings:
pattern: [{event_type: "ssp_corporate_action_informative_notification", event_subtitle: "Außerordentliche oder spezielle Hauptversammlung"}]
path: "Stocks/General Meetings/{iso_date_year}/"
filename: "{iso_date}.{iso_time} {event_subtitle}"
# Savings plan
savings_plan:
pattern: [{event_type: "SAVINGS_PLAN_INVOICE_CREATED"}, {event_type: "SAVINGS_PLAN_EXECUTED"}, {event_type: "SAVINGS_PLAN_CANCELED"}]
path: "Stocks/Savings plan/{iso_date_year}/"
# pre-determined tax base earning
stock_pre_earning_tax:
pattern: [{event_type: "PRE_DETERMINED_TAX_BASE_EARNING"}]
path: "Stocks/PreEarningTax/{iso_date_year}/"
# Dividends
dividends_received:
pattern: [
{event_type: "CREDIT", event_subtitle: "Dividende"},
{event_type: "CREDIT", event_subtitle: "Ausschüttung"},
{event_type: "CREDIT_CANCELED"},
{event_type: "ssp_corporate_action_invoice_cash"},
]
path: "Dividenden/{iso_date_year}/"
# Dividends Corporate action election
dividends_election:
pattern: [
{event_type: "INSTRUCTION_CORPORATE_ACTION", document_title: "Dividende Wahlweise(.\\d+)?"},
{event_type: "ssp_dividend_option_customer_instruction", event_subtitle: "Cash oder Aktie"},
{event_type: "ssp_corporate_action_informative_notification", event_subtitle: "Dividende Wahlweise"},
]
path: "Dividendelection/{iso_date_year}/"
# bonds
bond_repayment:
pattern: [{event_type: "REPAYMENT"}]
path: "Bonds/{iso_date_year}/"
filename: "{iso_date}.{iso_time} Repayment {event_title}"
bond_interest:
pattern: [{event_type: "COUPON_PAYMENT"}]
path: "Bonds/{iso_date_year}/"
filename: "{iso_date}.{iso_time} Interest {event_title}"
# Saveback
saveback_enabled:
pattern: [{event_type: "benefits_saveback_execution", document_title: "Enabled(.\\d+)?"}]
path: "Saveback/{iso_date_year}/"
filename: "{iso_date}.{iso_time} Enabled {event_title}"
saveback_executed:
pattern: [{event_type: "benefits_saveback_execution", document_title: "Abrechnung Ausführung(.\\d+)?"}]
path: "Saveback/{iso_date_year}/"
filename: "{iso_date}.{iso_time} Report {event_title}"
saveback_cost_report:
pattern: [{event_type: "benefits_saveback_execution", document_title: "Kosteninformation(.\\d+)?"}]
path: "Saveback/{iso_date_year}/"
filename: "{iso_date}.{iso_time} Cost report {event_title}"
# Round up
roundup_enabled:
pattern: [{event_type: "benefits_spare_change_execution", document_title: "Enabled(.\\d+)?"}] # same files - multiple times at once
path: "Roundup/{iso_date_year}/"
filename: "{iso_date}.{iso_time} Enabled {event_title}"
roundup_executed:
pattern: [{event_type: "benefits_spare_change_execution", document_title: "Abrechnung Ausführung(.\\d+)?"}]
path: "Roundup/{iso_date_year}/"
filename: "{iso_date}.{iso_time} Report {event_title}"
roundup_cost_report:
pattern: [{event_type: "benefits_spare_change_execution", document_title: "Kosteninformation(.\\d+)?"}]
path: "Roundup/{iso_date_year}/"
filename: "{iso_date}.{iso_time} Cost report {event_title}"
# account
cash_interest:
pattern: [{event_type: "INTEREST_PAYOUT"}, {event_type: "INTEREST_PAYOUT_CREATED"}]
path: "Cash Interest/"
filename: "{iso_date} Report"
cash_transfer_report:
pattern: [{event_type: "INCOMING_TRANSFER"}]
path: "Cash Report/"
filename: "{iso_date}.{iso_time} {document_title}" # {event_title} = Personal name
# annual tax report for account
account_tax_report:
pattern: [{event_type: "TAX_REFUND"},{event_type: "TAX_ENGINE_ANNUAL_REPORT"},{event_type: "YEAR_END_TAX_REPORT"}]
path: "Tax/"
filename: "{iso_date} {document_title}"
account_tax_adjustment:
pattern: [{event_type: "ssp_tax_correction_invoice"}, {event_type: "TAX_CORRECTION"}]
path: "Tax/"
filename: "{iso_date} {event_title}"
# common informations
notice_stocks:
pattern: [{event_type: "ORDER_CREATED", document_title: "Basisinformationsblatt(.\\d+)?"}]
path: "Notice/{iso_date_year}/"
filename: "{iso_date} {document_title} - {event_title}"
notice_stocks2:
pattern: [
{event_type: "TRADE_INVOICE", document_title: "Basisinformationsblatt(.\\d+)?"},
{event_type: "GESH_CORPORATE_ACTION", event_subtitle: "Unternehmensmeldung"}
]
path: "Notice/{iso_date_year}/"
filename: "{iso_date} {event_subtitle} - {event_title}"
notice_stocks3:
pattern: [{event_type: "CUSTOMER_CREATED"}]
path: "Notice/{iso_date_year}/"
filename: "{iso_date} {document_title}"
notice_option_contract:
pattern: [{event_type: "ORDER_EXECUTED", document_title: "Basisinformationsblatt(.\\d+)?"}]
path: "Notice/{iso_date_year}/"
filename: "{iso_date} {document_title} Option"
notice_multiple_documents:
pattern: [{event_type: "GESH_CORPORATE_ACTION_MULTIPLE_POSITIONS"}] #event_subtitle: Gesellschaftshinweis
path: "Notice/{iso_date_year}/"
filename: "{iso_date} {event_subtitle} - {event_title} - {document_title}"
contract_documents:
pattern: [
{event_type: "card_order_billed"}, # Bestellung Trade Republic Karte
{event_type: "DOCUMENTS_CREATED"}, # Basisinformationen über Wertpapiere
{event_type: "DOCUMENTS_ACCEPTED"}, # Rechtliche Dokumente: Kundenvereinbarung / Vorvertragliche Informationen / Datenschutzinformationen* / Widerrufsbelehrung* / *Crypto* / Risikohinweise
{event_type: "DOCUMENTS_CHANGED", section_title: "Dokumente"} # Rechtliche Dokumente: Kundenvereinbarung
]
path: "Contract/"
filename: "{iso_date} {event_title} - {document_title}"
contract_documents_updated:
pattern: [{event_type: "DOCUMENTS_CHANGED", section_title: "Aktualisierte Dokumente"}] # aktualisierte Rechtliche Dokumente: Kundenvereinbarung
path: "Contract/"
filename: "{iso_date} {event_title} - {document_title} updated"
here is the result.
in folder /Stocks/General Meetings/XXX/
before:
20xx-yy-zz.aa-bb Company AAA.pdf
20xx-yy-zz.aa-bb Company BBB.pdf
20xx-yy-zz.aa-bb Company CCC (5956d0ed-072c-48d0-8482-1fcd727df063).pdf
20xx-yy-zz.aa-bb Company CCC.pdf
20xx-yy-zz.aa-bb Company DDD.pdf
20xx-yy-zz.aa-bb Company EEE.pdf
20xx-yy-zz.aa-bb Company FFF (05cb8c83-7226-49fa-b2cd-b9ab11172158).pdf
20xx-yy-zz.aa-bb Company FFF.pdf
20xx-yy-zz.aa-bb Company GGG (d5a19969-e29f-43c9-aee8-756a27b7f27d).pdf
20xx-yy-zz.aa-bb Company GGG.pdf
20xx-yy-zz.aa-bb Company HHH.pdf
20xx-yy-zz.aa-bb Company CCC.pdf
20xx-yy-zz.aa-bb Company EEE (67812f93-1de7-4a42-9f1b-7f24e750b9b1).pdf
20xx-yy-zz.aa-bb Company EEE.pdf
20xx-yy-zz.aa-bb Company GGG.pdf
after:
20xx-yy-zz.aa-bb Company AAA.pdf
20xx-yy-zz.aa-bb Company BBB.pdf
20xx-yy-zz.aa-bb Company CCC - Hauptversammlung 1.pdf
20xx-yy-zz.aa-bb Company CCC - Hauptversammlung 2.pdf
20xx-yy-zz.aa-bb Company DDD.pdf
20xx-yy-zz.aa-bb Company EEE.pdf
20xx-yy-zz.aa-bb Company FFF - Hauptversammlung 1.pdf
20xx-yy-zz.aa-bb Company FFF - Hauptversammlung 2.pdf
20xx-yy-zz.aa-bb Company GGG - Hauptversammlung 1.pdf
20xx-yy-zz.aa-bb Company GGG - Hauptversammlung 2.pdf
20xx-yy-zz.aa-bb Company HHH.pdf
20xx-yy-zz.aa-bb Company CCC.pdf
20xx-yy-zz.aa-bb Company EEE - Hauptversammlung 1.pdf
20xx-yy-zz.aa-bb Company EEE - Hauptversammlung 2.pdf
20xx-yy-zz.aa-bb Company GGG.pdf
in folder /Contract/
before:
20xx-yy-zz Kundenvereinbarung (aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa).pdf
20xx-yy-zz Preis- und Leistungsverzeichnis (aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa).pdf
20xx-yy-zz Preis- und Leistungsverzeichnis.pdf
20xx-yy-zz Info zur Einlagensicherung (bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb).pdf
20xx-yy-zz Info zur Einlagensicherung.pdf
20xx-yy-zz Datenschutzinformationen (cccccccc-cccc-cccc-cccc-cccccccccccc).pdf
20xx-yy-zz Datenschutzinformationen.pdf
20xx-yy-zz Hinweise zu den Handelsplätzen (cccccccc-cccc-cccc-cccc-cccccccccccc).pdf
20xx-yy-zz Kundenvereinbarung (cccccccc-cccc-cccc-cccc-cccccccccccc).pdf
20xx-yy-zz Kundenvereinbarung.pdf
20xx-yy-zz Hinweise zu den Handelsplätzen (dddddddd-dddd-dddd-dddd-dddddddddddd).pdf
20xx-yy-zz Hinweise zu den Handelsplätzen.pdf
after:
202x-yy-zz Rechtliche Dokumente - Kundenvereinbarung.pdf
202x-yy-zz Rechtliche Dokumente - Preis- und Leistungsverzeichnis aktualisiert.pdf
202x-yy-zz Rechtliche Dokumente - Preis- und Leistungsverzeichnis.pdf
202x-yy-zz Rechtliche Dokumente - Info zur Einlagensicherung aktualisiert.pdf
202x-yy-zz Rechtliche Dokumente - Info zur Einlagensicherung.pdf
202x-yy-zz Rechtliche Dokumente - Datenschutzinformationen aktualisiert.pdf
202x-yy-zz Rechtliche Dokumente - Datenschutzinformationen.pdf
202x-yy-zz Rechtliche Dokumente - Hinweise zu den Handelsplätzen.pdf
202x-yy-zz Rechtliche Dokumente - Kundenvereinbarung aktualisiert.pdf
202x-yy-zz Rechtliche Dokumente - Kundenvereinbarung.pdf
202x-yy-zz Rechtliche Dokumente - Hinweise zu den Handelsplätzen aktualisiert.pdf
202x-yy-zz Rechtliche Dokumente - Hinweise zu den Handelsplätzen.pdf
in folder /Notice/ for general company announcements
before:
20xx-yy-zz Basisinformationen über Wertpapiere.pdf
20xx-yy-zz Basisinformationsblatt.pdf
20xx-yy-zz Basisinformationsblatt.pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben (eeeeeeee-eeee-eeee-eeee-eeeeeeeeeeee).pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben (ffffffff-ffff-ffff-ffff-ffffffffffff).pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben 1.pdf
20xx-yy-zz Kundenanschreiben 2.pdf
20xx-yy-zz Kundenanschreiben 1.pdf
20xx-yy-zz Kundenanschreiben 2.pdf
20xx-yy-zz Kundenanschreiben 1.pdf
20xx-yy-zz Kundenanschreiben 2.pdf
20xx-yy-zz Kundenanschreiben 3.pdf
20xx-yy-zz Kundenanschreiben 4.pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben.pdf
20xx-yy-zz Kundenanschreiben 1.pdf
20xx-yy-zz Kundenanschreiben 2.pdf
20xx-yy-zz Kundenanschreiben 3.pdf
20xx-yy-zz Kundenanschreiben 4.pdf
20xx-yy-zz Basisinformationsblatt.pdf
after:
202x-yy-zz Basisinformationen über Wertpapiere.pdf
202x-yy-zz Basisinformationsblatt Option.pdf
202x-yy-zz Basisinformationsblatt Option.pdf
202x-yy-zz Unternehmensmeldung - Company AAA.pdf
202x-yy-zz Unternehmensmeldung - Company BBB.pdf
202x-yy-zz Unternehmensmeldung - Company BBB.pdf
202x-yy-zz Unternehmensmeldung - Company CCC.pdf
202x-yy-zz Unternehmensmeldung - Company DDD.pdf
202x-yy-zz Unternehmensmeldung - Company CCC.pdf
202x-yy-zz Unternehmensmeldung - Company CCC.pdf
202x-yy-zz Unternehmensmeldung - Company DDD.pdf
202x-yy-zz Unternehmensmeldung - Company EEE.pdf
202x-yy-zz Unternehmensmeldung - Company FFF.pdf
202x-yy-zz Unternehmensmeldung - Company BBB.pdf
202x-yy-zz Unternehmensmeldung - Company CCC.pdf
202x-yy-zz Unternehmensmeldung - Company GGG.pdf
202x-yy-zz Unternehmensmeldung - Company BBB.pdf
202x-yy-zz Gesellschaftshinweis - Mehrere Investments - Kundenanschreiben 1.pdf
202x-yy-zz Gesellschaftshinweis - Mehrere Investments - Kundenanschreiben 2.pdf
202x-yy-zz Gesellschaftshinweis - Mehrere Investments - Kundenanschreiben 1.pdf
202x-yy-zz Gesellschaftshinweis - Mehrere Investments - Kundenanschreiben 2.pdf
202x-yy-zz Gesellschaftshinweis - Mehrere Investments - Kundenanschreiben 1.pdf
202x-yy-zz Gesellschaftshinweis - Mehrere Investments - Kundenanschreiben 2.pdf
202x-yy-zz Gesellschaftshinweis - Mehrere Investments - Kundenanschreiben 3.pdf
202x-yy-zz Gesellschaftshinweis - Mehrere Investments - Kundenanschreiben 4.pdf
202x-yy-zz Unternehmensmeldung - Company HHH.pdf
202x-yy-zz Unternehmensmeldung - Company AAA.pdf
202x-yy-zz Unternehmensmeldung - Company III.pdf
202x-yy-zz Unternehmensmeldung - Company JJJ.pdf
202x-yy-zz Unternehmensmeldung - Company KKK.pdf
202x-yy-zz Unternehmensmeldung - Company LLL.pdf
202x-yy-zz Unternehmensmeldung - Company JJJ.pdf
202x-yy-zz Gesellschaftshinweis - Mehrere Investments - Kundenanschreiben 1.pdf
202x-yy-zz Gesellschaftshinweis - Mehrere Investments - Kundenanschreiben 2.pdf
202x-yy-zz Gesellschaftshinweis - Mehrere Investments - Kundenanschreiben 3.pdf
202x-yy-zz Gesellschaftshinweis - Mehrere Investments - Kundenanschreiben 4.pdf
202x-yy-zz Basisinformationsblatt - Company BBB.pdf
I have created another Feature Request Issue #114 to further improve the file name,
before: 202x-yy-zz Gesellschaftshinweis - Mehrere Investments - Kundenanschreiben 1.pdf
.
after: 202x-yy-zz Gesellschaftshinweis - Company ZZZ - Kundenanschreiben 1.pdf
as well as other file names where the company name is not directly contained in the event block of the API response.
Just observed https://github.com/pytr-org/pytr/pull/112 we might lack an Event type "PAYMENT_INBOUND_GOOGLE_PAY". So i just transferred some € via credit card and Google Pay. I added "PAYMENT_INBOUND_GOOGLE_PAY"/"PAYMENT_INBOUND_CREDIT_CARD" and also tested it.
There might still be Apple Pay, but I don't have this to test it.
Sorry was in summer vacation. Let me do a check this week and then merge into the new version 0.3
Are we still interested in this, then i would resolve the conflicts again? If not, i would close the pull request.
Is your feature request related to a problem? Please describe. I would finally like a fixed folder structure, independent of API changes (like #91), which everyone can set according to their own preferences.
Describe the solution you'd like I suggest defining the target folders using several search criteria/search patterns. This search-target mapping should then be saved in a suitable configuration file. An example configuration file is attached.
for most search patterns you only need the following json nodes from the API-response: in root-event node: “type” ("event_type"), “subtitle” ("event_subtitle"), for a few documents you also need the “title” ("event_title"). In section-node: “title” ("section_title"), and finaly in "document"-node the “title” ("document_title").
most search patterns contain 1:1 strings. rarely, dynamic regex number patterns are also available, e.g. document number “\d” or for larger numbers e.g. years “\d+”.
There should also be a possibility for dynamic names in the target path, as predefined variables. In addition to the existing search patterns that can be reused in the path {event_type} - {event_title}, {event_subtitle}, {document_title}, there should also be environment variables, {iso_date} and {time} already exist, it would be nice to have the individual components, year: {%Y} => 2024, and of course month and day.
I personally find YAML format much more human-readable (especially because there is a comment option), nevertheless, in order to stay with the current technology and not want to introduce new dependencies into the project, here is the example configuration also in JSON format foldermap.json
Additional context
hope, this is possible.