[Bug]: Account Does Not Sync to SimpleFIN

weswitt commented 3 weeks ago

Verified issue does not already exist?

[X] I have searched and found no existing issue
[X] I will be providing steps how to reproduce the bug (in most cases this will also mean uploading a demo budget file)

What happened?

After adding a new account that connects to SimpleFIN I do not see any transactions or a correct balance. I can see that SimpleFIN is returning the correct data as I can run CURL to download the accounts from SimpleFIN and all the data are returned. When doing the "Bank Sync" I see the error "504 Gateway Time-out". I'm syncing to 7 different banks with many accounts so maybe AB is not waiting long enough since SimpleFIN returns all transactions for all accounts in one response.

Where are you hosting Actual?

Docker

What browsers are you seeing the problem on?

Chrome

Operating System

Windows 10

weswitt commented 3 weeks ago

OK, I enabled debug logging and I'm seeing this:

Error parsing JSON response: error code: 524 SyntaxError: Unexpected token e in JSON at position 0 at JSON.parse () at IncomingMessage. (file:///app/src/app-simplefin/app-simplefin.js:332:26) at IncomingMessage.emit (node:events:529:35) at endReadableNT (node:internal/streams/readable:1400:12) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) 2024-10-10T20:54:38.138Z info: POST 200 /simplefin/transactions

youngcw commented 3 weeks ago

Can you ping simplefin from your server running Actual?

weswitt commented 3 weeks ago

Yes. And as I said in the bug report I can download from SimpleFIN using a python script

weswitt commented 3 weeks ago

OK I debugged this further. I found that AB is issuing a request to SimpleFIN with the arguments "/accounts?start-date=1720828800&end-date=1730419200&pending=1", which a 4 month date range. This is likely a lot of data. So I used these same parameters in a python script and I found that what comes back is a response code of 524, timeout. The payload returned is HTML rendered by Cloudflare. This seems to be a problem with either Cloudflare or SimpleFN. I also observed that the timeout happened after 1m40.458s, so even if proper JSON data was eventually returned AB's 60 second timeout would have terminated the call.

@youngcw what do you think about this??

youngcw commented 3 weeks ago

The sync will always request 90 days of data unless you have less than that time in your account history. That 1m40s sounds like a long time to get data back. It should process on a few seconds.

weswitt commented 3 weeks ago

This is definitely a Cloudflare timeout, see https://developers.cloudflare.com/support/troubleshooting/cloudflare-errors/troubleshooting-cloudflare-5xx-errors/. Cloudflare will return a 524 after 100 seconds if the server does not respond. I am seeing the failure after exactly 100 seconds. This is absolutely caused by the fact that SimpleFIN is collecting a lot data by AB's request.

Actual Budget could fix this by making multiple calls for shorter date ranges and then combine the data.

weswitt commented 3 weeks ago

If I reduce the date range to one month it works fine.

weswitt commented 3 weeks ago

I modified getAccounts() in app-simplefin.js to never request more than 1 month and I now have things syncing again. This is not ideal because for new accounts it missed historical transactions.

I sent a support email to SimpleFIN to see if they can resolve this on their end.

youngcw commented 2 weeks ago

Did you ever hear back about this?

d4vastu commented 1 week ago

My issue is pretty odd. If I click bank sync some transactions do come in. However, if I want receive all new so actual matches my bank, I have to log in to simple fin, click adjust, wait for it to say my bank is already connected, then select bank sync again. Then everything works until the next day.

youngcw commented 1 week ago

My issue is pretty odd. If I click bank sync some transactions do come in. However, if I want receive all new so actual matches my bank, I have to log in to simple fin, click adjust, wait for it to say my bank is already connected, then select bank sync again. Then everything works until the next day.

@d4vastu Are you waiting until after simplefin has synced with your banks? Simplefin only syncs once a day so it can be up to 2 days before your transactions will show in Actual depending on the timing of things

weswitt commented 1 week ago

I did hear back from SimpleFIN. Their initial suggestion was the same as mine -- break up the sync calls into one month chunks. They sent another email back saying: "I've made a note to look into this, and I'll reach out to the Actual Budget folks, too. It looks like the timeout is happening on Cloudflare's side (and is by default set to 100 seconds). I'll see if we can extend that. Thanks again for bringing this up!".

Ultimately you could say this is a SimpleFIN bug, but I really think that AB needs to change their logic to make smaller requests so their code is defensive against these issues.

youngcw commented 1 week ago

Somewhat soon we will update the simplefin calls to only request a single account if you only request one. That probably will help your issue.

weswitt commented 1 week ago

That is a great solution and will definitely fix the problem. Thanks a lot!!

youngcw commented 1 week ago

That option wasn't available until a few days ago unfortunately, but its been added on simplefin's side so we can start using it now.

weswitt commented 1 week ago

Yea, I was going to ask about that as their documented API only returned data for all accounts. Great timing for this.

weswitt commented 1 week ago

I picked up your changes to app-simplefin.js. Seems to fix the problem, thanks.

Where did you find docs for the new SimpleFIN API to query for a single account?? I looked on the site and cannot find the docs.

youngcw commented 1 week ago

You may have to look at their github to see them as they are so new

actualbudget / actual