PSD2 with 2FA breaks the cronjob

[nodomain]

This is the error message.

/usr/local/lib/ruby/gems/2.4/gems/ruby_fints-0.0.3/lib/fints/dialog.rb:83:in `send_msg': {"9050"=>"Die Nachricht enthält Fehler.", "9075"=>"Starke Kundenauthentifizierung notwendig.", "9800"=>"Dialog abgebrochen", "9340"=>"Auftrag abgelehnt."} (FinTS::DialogError)
   from /usr/local/lib/ruby/gems/2.4/gems/ruby_fints-0.0.3/lib/fints/dialog.rb:67:in `init'
   from /usr/local/lib/ruby/gems/2.4/gems/ruby_fints-0.0.3/lib/fints/client.rb:20:in `get_sepa_accounts'
   from /root/ynab-bank-importer/lib/dumper/fints.rb:21:in `fetch_transactions'
   from /root/ynab-bank-importer/lib/account.rb:18:in `fetch_transactions'
   from run.rb:11:in `block in <main>'
   from run.rb:9:in `map'
   from run.rb:9:in `<main>'

Any idea how to deal with this?

[gitviola]

I fear there is no way to deal with this for now 😞

I'm trying to find a service that allows us to access transaction data for free but so far I didn't find any that fits the needs. All PSD2 compliant offerings out there are focused on companies who can effort to acquire the licence/certificate that is apparently needed to use bank apis. I'm very disappointed by the choice to leave out individual contributors.

[manuelgrabowski]

Just for clarification: Implementing the required 2FA would require a license/certificate that costs money or effort? I was under the impression that it would "just" be annoying, not basically impossible for non-commercial software, sigh.

[PetePriority]

I stumbled upon python-fints while looking for alternatives. In their docs they mention

As of September 14th, 2019, all FinTS programs should be registered with the ZKA or banks will block access. You need to fill out a PDF form and will be assigned a product ID that you can pass above.

If you set the product_id to None, the library will fill in the default product ID for python-fints. This works fine for evaluation, but should never be used if you bundle python-fints within a larger project. This might also not be acceptable by some banks.

Click here to read more about the registration process.

I'm not sure if they'd issue these product IDs for open source projects since I assume they're supposed to be kept secret?

[gitviola]

@manuelgrabowski from what I've seen a certificate/license is needed to directly use the APIs of banks. However, there are some services like token.io (provider that N26 partners with for the PSD2 compliance) and some others, but they all seem to charge money. Friday I found out that TrueLayer.com is doing a Germany beta, so I started contacting them that day and will try to see if that could work for us. But to summarise my feeling: I think it will take some time until there are providers with free offers that can be used for offline access

@PetePriority I actually have a registration number (which I haven't implemented here yet). I could give it a try in the next few days. But I don't think that it will fix the 2-factor issue. What we need is a way to get permanent "offline access" (access without the need of any action when the script runs) and from what I understood, that's what we need a certificate/licence for (or use a service that has the certificate/licence)

[117agu]

Does anybody know if there are there plans of ynab to support PSD2 directly?

[gitviola]

@117agu I wrote them an email today so that they are aware that even the maintainers of import-scripts are frustrated and can't do much about it. I think they relied on the community strength to get this figured out. Obviously, their main market is north america and it takes quite some effort and resources to understand the financial system of an other economic system. But they need to understand that the first solution that solves this issue will win all those users who have our need of automating the import of transactions. So if other companies like Mint (competitor to YNAB) solves this issue first, I'm sure that they will win quite some YNAB EU users.

I don't think they're actively working on a solution for EU users. In the support forums it says that they're passively watching their import-partners to support EU banks. In my opinion they should rather look for a new partner focused on the EU market as a short-term solution until their current providers add support for EU institutions.

[DWegmann]

As a workaround for my YNAB imports I currently using the OFX export functionality of the money money Mac app (https://moneymoney-app.com/). They are also currently working hard to make their account access working with PSD2.

[manuelgrabowski]

@schurig Thanks for the detailed explanation and your efforts finding a solution. I could even live with a TAN-based method, because I will change my bank to one that doesn't require a TAN more than once a month anyway – and that's something that could be dealt with gracefully enough in a cronjob as well, I think. I'm okay with some manual commands once a month to keep it running.

@117agu Very doubtful, they repeatedly made it clear that they have no interest in applying their own resources here. The US banks they support are only supported by a third-party provider as well, and they stated they don't want to make that more complex by supporting more than one provider. Maybe this stance will change in the PSD2 aftermath, but I doubt it – officially they don't even condone importing the bank statements automatically anyway.

[gitviola]

@manuelgrabowski I'm currently looking at bunq which is a dutch bank. They have an API which looks promising and reading their docs, you can still use the API without a licence (the api it's part of their product offer). The only downside is a monthly fee of ~8€. But I'll wait for TrueLayer's Germany beta to see if their service might workout before I try around with bunq.

[117agu]

Damn, the only solution might be to create a European ynab clone ;(

[manuelgrabowski]

TrueLayer looks quite interesting. I actually wouldn't even really mind a monthly fee for a bank account with good features, but I'll wait a little until all the dust has settled. I was shocked to learn that N26 doesn't even allow to do wire transfers via HBCI but forces me to use the website instead of the trusted MoneyMoney app. So in the next months I'll decide where to go – a bank with sane PSD2 implementation, proper HBCI support and ApplePay.

On a personal, slightly off-topic tangent: I think this is the final nail in the coffin for YNAB and myself. While I understand their reasoning about not wanting to add complexity to their core product for a feature they actually kinda discourage you from using at all, I'm still a bit frustrated by their strong US-focus. What's more is that they didn't really improve the core product in meaningful ways for me during the whole last year. There is a lot of little QoL-improvements they are not doing, some that are still missing from the old YNAB desktop app, some that are "just" missing between current web and iOS app. (For example on iOS, typing "Ba" will suggest "Bäcker" as autocompletion, while on the web app, only typing "Bä" will find that result.) I've been rather using it for after-the-fact book keeping than actual budgeting, and now with additional manual effort it just doesn't feel worth it for me anymore, especially not with a recurring usage fee.

I'll definitely still follow the developments here out of technical curiosity, though. 🙂

[Password2501]

@schurig thanks for providing your service in the first place it helped me a lot when I moved from France (nice Quicken export Banks) to germany (crappy excel export). But now I'm fucked, I can't keep my budjet up manually. and as @manuelgrabowski says It might be the nail on YNABs coffin...

@schurig keep us posted on your work, may be we can find enough people to pay for some nice implemtation/licence...

[gitviola]

I have some news. So I couldn't wait to try it and I subscribed to bunq. They offer an api as their product. And I saw that they support webhook-notifications which means that I don't even have to put my bank credentials or access token anywhere. I just needed it once on whatever computer to setup the webhook. It's pointing to my uberspace where I wrote a small nodejs script which takes the payload, builds the transaction and creates it in YNAB. Now I pay and it's there instantly. I wish other banks would offer this kind of service. Now I have a bank account that works extraordinary well for the YNAB import and 3 banks that don't work at all. Sadly this costs 8€ + my uberspace. Instead of the uberspace I could have used my raspberry pi and set up DynDNS so that I have a domain pointing to it + port-forwardings. However, there is a higher chance that this is not available 24/7, so I picked the uberspace. I couldn't find anything about a retry-mechanism when they send out the webhooks.

In case anyone is interested, I posted the small nodejs express server here: https://gist.github.com/schurig/2f9b97587804d0410097d5488ffc812a

[wizonesolutions]

Just to clarify, as there is the label but not an explicit mention: this affects N26 as well, yeah? It's a shame. PSD2 was supposed to expand access to this kind of data, not break existing methods...

Well, at least N26 has CSVs, and there's https://aniav.github.io/ynab-csv/. Not great and going to miss the automation for now. Will follow this issue.

[gitviola]

@wizonesolutions yes, the same thing is happening with N26.

Thanks for posting that link! I actually do the same those days until there is a better solution. I tried TrueLayer's Germany beta but not all banks work yet (unluckily none of my banks works so far). They're still working on the N26 integration for example. As soon they support all German banks I will continue my efforts on that front.

[f2cx]

I have successfully used Salt Edge API to connect to the German bank "DKB" to have the import working again. Salt Edge can connect to hundreds of PSD2 banks, so this solution might also work for others. I could just register for an account for free. You need to request "test access" after account creation, so that you can talk to real banks.

Of course, this will still stop fully automated cronjobs, because DKB asks for a new TAN everytime (not only every 90 days). So I just sync my transactions with Salt Edge semi-automatic using a curl script, which is sending me to a website provided by Salt Edge to enter login data and TAN.

Afterwards I can just run "ynab-bank-importer" to import the transactions (including all accounts and credit cards). The login to the Salt Edge API does not require 2FA.

It is not heavily tested and more a personal proof of concept, if this might be a viable solution for my case. The code for the dumper is here: https://github.com/schurig/ynab-bank-importer/pull/76

[gadmeer]

@f2cx wow! Thanks for finding out. Can you help to clarify if Salt Edge mean by 100 live connections 100 bank accounts or transactions? Is it still working on your side?

[f2cx]

@gadmeer It means 100 connections to different banks. However, currently I only sync to the German bank "DKB" (I have imported thousands of transactions on multiple accounts of that bank) and to Paypal. So in total I use only 2 live connections.

Currently, I'm using this Salt Edge account only for my own purposes, so the 100 live connection quota is obviously not really an issue. However, I have no idea if Salt Edge might stop supporting such test accounts at some point in the future, which would be really sad. I really appreciate their generous price model to have free accounts for such testing purposes.

Everything works in general. Sadly, there is an annoying issue with my specific bank. In my case for the DKB connection, they cannot split "payee name" and "description" of a transaction. So currently the payee name of my imports are always empty and the description contains both payee name and description as one field. There is no reliable separator in between, so currently I could only use heuristic approaches to guess the payee name, which I haven't implemented yet.

However, this is not a general Salt Edge issue, it seems to be a specific issue for the DKB connection provider. So it might still perfectly work for your bank!

By the way, I have no idea, why YNAB themself doesn't partner with Salt Edge to officially use their services. There is an older discussion about this topic, in which other users (and even an employee of Salt Edge) specifically suggest that YNAB should use Salt Edge to have direct import for international customers. However YNAB already seem to have two other direct import partners and YNAB is reluctant to add any other: https://support.youneedabudget.com/t/h4p8m9/open-banking-api

[Putr]

FYI this: https://github.com/guitmz/n26

works for N26 - well at least for current balance and some other endpoints, it's throwing an error on transactions.

Anyway when you run the script the N26 app asks for confirmation, if you do it in 60 seconds it lets the script through. There's even a PR to cache the token for longer.

[dequis]

For n26, https://github.com/femueller/python-n26 handles 2FA, can store the refresh token, but it requires reauthentication after two hours: https://github.com/femueller/python-n26/issues/67

People on that thread contacted n26, and got told that they could use token.io.

For now I'm making it slightly-less-manual than exporting CSV from the web UI and using this unholy commandline to turn the json to CSV:

n26 -json transactions --limit 100 --from 2019-11-15 --to 2019-12-08 > transactions.json
cat transactions.json | jq -c '.[] | [(.visibleTS / 1000 | localtime | strftime("%Y-%m-%d")), .merchantName // .partnerName, .referenceText // "", .amount|tostring]' | tac | sed 's/^\[//; s/\]$//; 1 {s/^/"Date","Payee","Memo","Amount"\n/}' > transactions.csv

For me the steps ahead are:

1. Minimize the effort of doing manual imports (with my awful commandline, the ynab part is manual)
2. Intercept 2FA responses somehow. The python library allows switching mfa_type from "app" to "sms", and there's plenty of ways to access a phone's SMS remotely (e.g. http://projectmaxs.org but I would prefer not to have to deal with XMPP)

EDIT: I turned my manual imports procedure into a python script, if anyone is curious: https://gist.github.com/dequis/6b510cda124bd7a2ce9430339ea9f33a

[zappingseb]

Thankfully, at least comdirect has its own API now. As a starter I created a repo to work with it. It cannot be executed on a raspberry, yet, as a TAN needs to be solved. https://github.com/zappingseb/comdirect_api

[stereolith]

The python-fints lib seems to have solved PSD2 integration. I think an interactive command line utility is used to enter a TAN. Docs

I wrote a YNAB importer with this lib and it works well for me. My bank does not require a TAN to retrieve transactions, so I could not test TAN handling yet. Here is the repo if you're interested.

[silasalberti]

There currently is a survey by the ECB about the implementation of the digital euro. The following question immediately reminded me of this issue:

What requirements (licensing or other) should intermediaries fulfil in order to provide digital euro services to households and businesses? Please base your answer on the current regulatory regime in the European Union.

The problem in the case of this repo (and probably many other cases) was that PSD/2 killed Open Source projects by requiring expensive licenses for bank integration that indie/open source developers can't afford. (Another example: There also was a very cool project that integrated your bank account with smart home devices, like flashing a light bulb if your account balance is low or whatever you imagine. Very sad imo that all this is broken now.) In my replies I made them aware of this issue, hoping that they'll make the future APIs as accessible as possible. However, it will of course have a bigger impact if even more people raise awareness on this issue. Hopefully, they'll take notice and make the "digital euro" accessible without expensive & bureaucratic licenses, maybe by giving a limited entry tier for small projects :)

TL;DR: This is the chance to have our voices heard.

[emeni09]

in the end is the n26 dumper working now? or still unsuported? I get "Couldn't login with your provided N26 credentials. Please verify that they're correct. (RuntimeError)" but after checking my login credential 3 times I am quite sure something else is happening :(