[Feedback]: SimpleFIN Sync

[zachwhelchel]

This issue is to track thoughts/bugs/concerns/ideas around the new SimpleFIN syncing which is an experimental feature.

[JMLegere]

How do I use this?

[youngcw]

@JMLegere Currently you have to be using the edge release, then enable the experimental feature for it

[sethgillett]

Any concerns about how actively developed their bridge tool is? Their last blog post is from 2016 and their terms of use haven't been updated since 2020, and it doesn't seem like any banks have implemented the protocol themselves.

Edit: It looks like their privacy policy was updated last October.

[youngcw]

@sethgillett SimpleFIN uses MX in the background and its primary role is to make bank sync accessable to individuals instead of having to use a large company service. So there likely isn't much that needs changed. From what I've heard the guy that runs it is very responsive and actively keeping it going.

[yoyotogblo]

He's very responsive. Almost always responds within a few hours of issues being raised and escalates issues quickly also. I won't be concerned about it being active.

[joel-jeremy]

We should add a new feature to "force match" an existing transaction to a newly imported one. This should basically just assign the imported_id of the new imported transaction to an existing transaction. This way when the next sync runs, the existing transaction will be matched and will not be duplicated.

[yoyotogblo]

We should add a new feature to "force match" an existing transaction to a newly imported one. This should basically just assign the imported_id of the new imported transaction to an existing transaction. This way when the next sync runs, the existing transaction will be matched and will not be duplicated.

That's how it already works. If the imported_id exists, it auto-matches so doesn't dupe. If it doesn't exist, it uses fuzzy matching (looks at transactions 4 days prior and 1 day after the imported transactions) and if it finds one, it assigns that existing the transaction the imported_id.

[joel-jeremy]

I'm aware of that, and what I meant is for users to be able to do what the fuzzy matching is doing by manually selecting which transactions to "match". Because the fuzzy matching will not be able to catch all the cases - in my case, the bank seems to use the cleared date as the transaction date instead of the date the purchase was made and that varies between 1 day or sometimes a week.

Another thing I noticed is that the imported transactions doesn't seem to match split transactions.

[cmccambridge]

The fuzzy logic (-4 to +1 days) makes sense when syncing of all accounts is happening on an on-going daily basis, and we hope that most transactions appear in chronological order across acounts. But, it's been a problem tonight as I'm trying out new SimpleFIN links to existing Actual accounts...

Is there a reason to keep the match range asymmetric? Could it be updated to -4 to +4, for example? Else definitely the "force match" concept that @joel-jeremy mentioned would resolve my case as well... I had gone looking for that exact feature at first, but couldn't find it.

Here's the problematic scenario for my data:

1. Local checking account existed. Local credit card account existed.
2. On a past date (typically my problem is near a weekend when credit card and bank disagree on effective dates), a payment was made from my checking account to the credit card.
3. Previously, using QXF imports, the Checking account recorded the transfer on, for example, Nov 6, 2023 (a Monday). The credit card recorded it on Nov 3, 2023 (a Friday).
4. I dealt with the duplicate/non-matched transactions by deleting one or the other, depending which account I happened to import via QXF first to trigger the transfer Rules.
5. In future updates, I avoided the problem by using "last 30 days" type QXF files to dodge the duplicates
6. When I linked the checking account to SimpleFIN, all was well: Fuzzy match worked and the payment on Nov 6 looked good from the checking account side.
7. When I linked the credit card to SimpleFIN, it imported the Nov 3 payment again, and the fuzzy match failed, presumably since the real transfer on Nov 6 was too far into the future.
8. I deleted the new Nov 3 payment manually, synced again, and the SimpleFIN sync recreated it ☹️.

I think the better workaround as I identify these date collisions is for me to go change the dates of the transfer to try and favor the earlier date?

But, the manual work to figure out which account I need to forcibly tweak is overhead that would be great to avoid if possible 😄

[yoyotogblo]

I'm aware of that, and what I meant is for users to be able to do what the fuzzy matching is doing by manually selecting which transactions to "match". Because the fuzzy matching will not be able to catch all the cases - in my case, the bank seems to use the cleared date as the transaction date instead of the date the purchase was made and that varies between 1 day or sometimes a week.

Another thing I noticed is that the imported transactions doesn't seem to match split transactions.

Ah yeah, manually matching...

I wrote a lot of details about the ideal behavior in the issue below... Fully agree that we need a way to manually match. https://github.com/actualbudget/actual/issues/669

Also, on the 1 day vs a week time difference, I have a PR.to change the fuzzy matching logic to 7 days prior and 7 days after. I use that in my instance and it works much better. https://github.com/actualbudget/actual/pull/2300

[yoyotogblo]

The fuzzy logic (-4 to +1 days) makes sense when syncing of all accounts is happening on an on-going daily basis, and we hope that most transactions appear in chronological order across acounts. But, it's been a problem tonight as I'm trying out new SimpleFIN links to existing Actual accounts...

Is there a reason to keep the match range asymmetric? Could it be updated to -4 to +4, for example? Else definitely the "force match" concept that @joel-jeremy mentioned would resolve my case as well... I had gone looking for that exact feature at first, but couldn't find it.

Here's the problematic scenario for my data:

1. Local checking account existed. Local credit card account existed.
2. On a past date (typically my problem is near a weekend when credit card and bank disagree on effective dates), a payment was made from my checking account to the credit card.
3. Previously, using QXF imports, the Checking account recorded the transfer on, for example, Nov 6, 2023 (a Monday). The credit card recorded it on Nov 3, 2023 (a Friday).
4. I dealt with the duplicate/non-matched transactions by deleting one or the other, depending which account I happened to import via QXF first to trigger the transfer Rules.
5. In future updates, I avoided the problem by using "last 30 days" type QXF files to dodge the duplicates
6. When I linked the checking account to SimpleFIN, all was well: Fuzzy match worked and the payment on Nov 6 looked good from the checking account side.
7. When I linked the credit card to SimpleFIN, it imported the Nov 3 payment again, and the fuzzy match failed, presumably since the real transfer on Nov 6 was too far into the future.
8. I deleted the new Nov 3 payment manually, synced again, and the SimpleFIN sync recreated it ☹️.

I think the better workaround as I identify these date collisions is for me to go change the dates of the transfer to try and favor the earlier date?

But, the manual work to figure out which account I need to forcibly tweak is overhead that would be great to avoid if possible 😄

I have a PR to make it symmetrical and to use 7 days in both cases... https://github.com/actualbudget/actual/issues/2300

I had the same issues as you with the Simplefin sync and this has solved it. BTW, currently, the Simplefin sync looks back 90 days. Most accounts don't provide that much history so you won't see as many breaks. But for the ones that do, it could break over a 90 day range everytime you sync.

[joel-jeremy]

Another observation: This is not SimpleFIN specific but when you modify an imported transaction, it will still match the next time you sync even though the fields are no longer the same. I think we should treat imported transactions like we do reconciled transactions where a warning pops up when editing an imported transaction.

[cmccambridge]

This one might be SimpleFIN specific, and might also not be fixable depending what data we have to work from, but I'll ask :)

Is there any additional metadata we can present on the Link UX flow to help clarify what's what in the available SimpleFIN connections? In my connections, some appear to use the "friendly name" i set through the bank UI, or a common name for the card, but others are very generic and I could only figure out which account it was through process of elimination. On one I even had to guess, realize it was wrong, delete the imported transactions, and try again :)

Is there any connection name we could show? Or a partial account number to disambiguate multiple similar-named accounts?

[MatissJanis]

👋 I tried setting up simplefin to run some tests. Here are some issues I encountered:

• [ ] we definitely need a docs page; similar to the one we have for gocardless
• [ ] there is no way to set a new token (initially I used an invalid token; this causes the /accounts request to load forever without any way to replace this invalid token with something else); -- maybe there actually is a way to update the token.. but I'm not able to get to it
• [ ] could we add a "use test credentials" button that uses the test secret (from here) if folks are running a dev server? Would make testing it much quicker
• [ ] it seems that the notes field is always the same as payee field: is that expected (I'm using the above test credentials)?

Everything else seems to be working good so far. Great job!

[swelborn]

Hello! It may be good to have a marker to designate transactions that were imported through syncing with simplefin. I may be missing this if it is there. Ideally, one could toggle that marker off/on depending on if they wanted it there or not.

all around awesome and seamless. Need to do a bit of rule updating, but other than that all of my accounts are synced up. great work.

[joel-jeremy]

It may be good to have a marker to designate transactions that were imported through syncing with simplefin

This is a good idea. We mark linked/imported accounts in the UI and we should do the same with imported transactions (gocardless/simplefin) IMO. This + warn users if an imported transaction is being modified. If modified, it should then be converted to a regular transaction (we remove it's imported_id).

[yoyotogblo]

Another observation: This is not SimpleFIN specific but when you modify an imported transaction, it will still match the next time you sync even though the fields are no longer the same. I think we should treat imported transactions like we do reconciled transactions where a warning pops up when editing an imported transaction.

This is how I'd expect it to work though. It matches the imported id and since it recognizes that you've already imported that transaction and edited it, it shouldn't try to change it and should just match it. Not sure of the benefit of a warning pop up in this instance.

[yoyotogblo]

This one might be SimpleFIN specific, and might also not be fixable depending what data we have to work from, but I'll ask :)

Is there any additional metadata we can present on the Link UX flow to help clarify what's what in the available SimpleFIN connections? In my connections, some appear to use the "friendly name" i set through the bank UI, or a common name for the card, but others are very generic and I could only figure out which account it was through process of elimination. On one I even had to guess, realize it was wrong, delete the imported transactions, and try again :)

Is there any connection name we could show? Or a partial account number to disambiguate multiple similar-named accounts?

There's definitely more info that can be shown... If you use the external test script that Dup created, that has more info. I ran into the same thing and played the same guessing game and deleting transactions. Good thing is you only have to do it one time :)!

[yoyotogblo]

It may be good to have a marker to designate transactions that were imported through syncing with simplefin

This is a good idea. We mark linked/imported accounts in the UI and we should do the same with imported transactions (gocardless/simplefin) IMO. This + warn users if an imported transaction is being modified. If modified, it should then be converted to a regular transaction (we remove it's imported_id).

Definitely don't think we should ever be removing the imported_id. Based off your proposal, if it imports a transaction and I need to change the payee, then that's now modified. Remove the imported_id and the transaction gets duped. Not sure I see any benefit to what you're proposing.

[swelborn]

In terms of the marker, I was only thinking of a label like your green dots next to the linked accounts. I don't think this would involve removing a uid as suggested above, but adding a tag... but I'm not familiar with source so I'm not sure how difficult this would be.

I am checking actual from mobile now and there is no sync button. I understand that the mobile version is not as fully featured as desktop, but a sync button would be good for mobile.

[joel-jeremy]

This is how I'd expect it to work though. It matches the imported id and since it recognizes that you've already imported that transaction and edited it, it shouldn't try to change it and should just match it. Not sure of the benefit of a warning pop up in this instance.

I am referring to user-made edits not edits made by the importer if that's what your thinking.

Definitely don't think we should ever be removing the imported_id. Based off your proposal, if it imports a transaction and I need to change the payee, then that's now modified. Remove the imported_id and the transaction gets duped. Not sure I see any benefit to what you're proposing.

The warning popup can be limited to amount user edits or possibly even the date. It doesn't make sense to have an imported transaction's amount be different from the amount that came from the bank. It can also introduce confusion when users need to reconcile old transactions.

[joel-jeremy]

I am checking actual from mobile now and there is no sync button. I understand that the mobile version is not as fully featured as desktop, but a sync button would be good for mobile.

You should be able to pull down on the account page to sync.

[yoyotogblo]

I am referring to user-made edits not edits made by the importer if that's what your thinking.

I make user edits all the time. Quite often, I'm changing the notes. At times, I'm creating split transactions. Or manually editing the payee. I don't see a scenario where I'd have edited an imported transaction, and would then want the same transaction imported again with the original details (which is what would happen if the imported ID is removed).

Can you explain what goal you're trying to accomplish?

[joel-jeremy]

I make user edits all the time. Quite often, I'm changing the notes. At times, I'm creating split transactions. Or manually editing the payee.

Notes and payees are fine. It's just the amount (or possibly even the date) that need to be kept consistent with the original imported data that came from the bank. Changing the amount essentially makes it a totally different transaction and could introduce confusion when needing to reconcile down the line.

Essentially what I am getting at it to make imported transactions as reconciled transactions but only warn when editing the amount. This is to make it clear that you are overriding the data that came from your bank. However, the warning popup should still allow the user to choose whether to retain the imported status of the transaction or remove it IMO.

[yoyotogblo]

Definitely not the date. That would create a disaster when there are transfers. Pretty much always, a payment to a credit card or transfer between bank accounts would have different dates. So the already imported transaction could end up with a different date than the new imported transaction and you still would want them to match.

So that leaves only amount as the use-case. I agree people shouldn't be modifying amounts on transactions. That's really the only scenario I see a warning being useful. But if a person modifies the amount on an imported transaction, not sure reimporting the transaction is the way to go. But I can see that argument though. A warning though would be useful.

[crobibero]

I linked my checking, savings, and credit card successfully. My checking and saving came in fine but my credit card has the transactions inverted. Maybe we need a way to designate an account as inverted? I don't see anything in the SimpleFin response that would indicate whether an account is a credit card.

[prizmpaladin]

Beginner here. I'm running Actual on Unraid using a docker from https://hub.docker.com/r/actualbudget/actual-server. How would I go about getting the edge release to use this feature?

[yoyotogblo]

It's in the latest release so you don't need edge. Use these instructions to update to the most recent version of Actual (https://actualbudget.org/docs/install/docker/).

Once you're up to date, you need to enable it under Settings -> Experimental Features

[prizmpaladin]

It's in the latest release so you don't need edge. Use these instructions to update to the most recent version of Actual (https://actualbudget.org/docs/install/docker/).

Once you're up to date, you need to enable it under Settings -> Experimental Features

Slick, just got this working. Thanks for pointing me in the right direction.

[culpeppers]

I appreciate the work on this!

Is anyone else having an issue with their Discover accounts not showing in Actual? It is linked fine in SimpleFIN, and the rest of my linked accounts are showing but not Discover.

[prizmpaladin]

I appreciate the work on this!

Is anyone else having an issue with their Discover accounts not showing in Actual? It is linked fine in SimpleFIN, and the rest of my linked accounts are showing but not Discover.

I just added my discover accounts and am seeing the same behavior.

[newmanwj]

I appreciate the work on this! Is anyone else having an issue with their Discover accounts not showing in Actual? It is linked fine in SimpleFIN, and the rest of my linked accounts are showing but not Discover.

I just added my discover accounts and am seeing the same behavior.

Same here with Discover. Thought it wouldn't pick up any new accounts, but AmEx cards showed up fine.

[culpeppers]

I stood up a local copy and tested it out. It looks like it's SimpleFIN that is not returning the accounts and nothing to do with the implementation on Actual.

I submitted a report to SimpleFIN about this, will update when I hear back.

[kwispykweems2]

I appreciate the work on this! Is anyone else having an issue with their Discover accounts not showing in Actual? It is linked fine in SimpleFIN, and the rest of my linked accounts are showing but not Discover.

I just added my discover accounts and am seeing the same behavior.

Same here with Discover. Thought it wouldn't pick up any new accounts, but AmEx cards showed up fine.

I have two Discover CC accounts, and I setup the first just fine. When I setup the second in SimpleFIN, Actual sync UI doesn't even see the second one. My first is called "Account", quite descriptive. ;)

[cgiacofei]

Syncing is great so far. One thing I miss from the standalone script is the option to have it modify the account notes with current balance given by the bank. I find that really useful for doing a quick reconciliation check.

[latetedemelon]

@duplaja has an alert script about the sync status of the accounts you've added to your simplefin account. Incorporating something like this would be nice in a future version

https://github.com/duplaja/simplefin-alerts

(Overall apprise integration in general would be great too).

[culpeppers]

I heard back from SimpleFIN that they submitted tickets with MX and will let me know once they've heard back.

[latetedemelon]

I believe @iffy accepts issues via the github repo for his website:

https://github.com/simplefin/simplefin.github.com

[culpeppers]

Regarding matching transactions, I encountered a use case that probably needs its own special handling

When a transfer transaction is imported, it arrived as two distinct transactions. Since, at least in my budget, it doesn't automatically know to turn them into a single transfer. So when I add one side, it creates a transfer transaction. However, the other end of the imported transaction cannot be matched. If I delete it, it comes back since it's not added. If I turn that into a transfer and delete the other one, that transaction returns.

[youngcw]

@culpeppers Have you made rules for both sides of the transfer? The Docs give an example of using rules to dedup imported transfers.

[karmanbadhesha]

This window should definitely show the financial institution and the balances. For example, if you have multiple mastercards it's very difficult to tell which one it is

[faultoverload]

Just a small issue I noticed with the Link Accounts page. No scroll bar is created for the drop down with "Account in Actual" so you cant scroll through the list since it scrolls the whole page. Not a breaking issue since you can just search for the account you want.

I also noticed that when first going through the linking process it would forget my Simplefin token after opening and closing the link accounts menu a few times. Im not sure if this is some rate limiting thats my fault or something else since like many others I was linking accounts one at a time to figure out which one was which.

[blampe]

I set this up last night and the experience so far has been good! US bank sync was the biggest blocker preventing me from using Actual and I'm very happy to see it working now. Below are some first impressions / wishlist items -- none of these are showstoppers.

• [ ] Ability to add SimpleFin accounts as off-budget. (Or the ability to make accounts off-budget after adding them -- maybe it's already there and I'm not seeing it.)

I'd like to add my partner's Venmo account to make it easier to categorize some withdrawals from checking (the Venmo transactions include more helpful notes), but I don't want to have to categorize or deal with transfers to the Venmo.

• [x] Better handling of ACH settlements.

Some payments can take a few days to settle and these don't always get correctly matched as transfers. I think #2300 addresses this but I haven't tested it.

• [ ] Daily server-side sync.

I gather from https://github.com/actualbudget/actual-server/pull/296 that this will only sync transactions when the client initiates. I have a lot of accounts, and I don't want to have to wait for things to sync when I'm trying to quickly categorize weekly spending.

• [ ] A way to visually group accounts in the UI.

SimpleFin enabled me to add all of my accounts for an awesome birds-eye view of everything. I've resorted to naming accounts with emojis to better differentiate them in the sidebar 😆

[youngcw]

@blampe A few of the things you listed can be handled quite well using the Rules engine along with #2300 like you mentioned. The docs say how to handle synced transfers with Rules.

You should be able to like off budget accounts. You need to manually link an account that already exists in Actual using the "Link Account" option in the account menu.

I believe you can sync all your accounts by using the sync button in the all accounts view.

[Amolith]

@MatissJanis:

• [ ] there is no way to set a new token (initially I used an invalid token; this causes the /accounts request to load forever without any way to replace this invalid token with something else); -- maybe there actually is a way to update the token.. but I'm not able to get to it

— https://github.com/actualbudget/actual/issues/2272#issuecomment-1922232646

I deleted the app from SimpleFIN's side and generated a new token without realising there's currently no way to change it from Actual's side.

I do see a secrets table in the database (server-files/account.sqlite) with simplefin_token and simplefin_accessKey, but I don't know whether deleting those rows or changing simplefin_token would break other things.

[duplaja]

@Amolith

I do see a secrets table in the database (server-files/account.sqlite) with simplefin_token and simplefin_accessKey, but I don't know whether deleting those rows or changing simplefin_token would break other things.

This is the way to do it for now. Stop Actual Server, delete just those two rows from account.sqlite in the secrets table, and then can start Actual Server back up again.

[LinuxTeamster]

I just saw this as an option and set it up, it was pretty easy and straightforward with the documentation. I use two small Credit Unions and they were both supported and imported with no problems. I love it, great work

[duplaja]

Once this moves out of experimental, it might be worth seeing about getting Actual added to SimpleFIN's Apps / integrations page: https://www.simplefin.org/ecosystem.html

[Amolith]

delete those two rows from account.sqlite , secrets table, and then can

@duplaja I mistakenly read this as "delete those two rows from account.sqlite, [delete the] secrets table, and then...". My secrets table is now gone and I'm not familiar enough with actual's codebase to figure out what the schema is to recreate it.

Would someone with a working setup mind pasting the table schema here?

# sqlite3 server-files/account.sqlite 'pragma table_info(secrets);'

[duplaja]

@Amolith

Would someone with a working setup mind pasting the table schema here?

0|name|TEXT|0||1
1|value|BLOB|0||0

I'll edit the original instructions to make more clear.