[Bug]: Transaction not synced with Gocardless, has no transactionId

Verified issue does not already exist?

[X] I have searched and found no existing issue

Is this related to GoCardless, Simplefin or another bank-sync provider?

[X] I have checked my server logs and could not see any errors there
[X] I will be attaching my server logs to this issue
[X] I will be attaching my client-side (browser) logs to this issue
[X] I understand that this issue will be automatically closed if insufficient information is provided

What happened?

After syncing bank transactions with Gocardless, I noticed that one transaction was missing due to the balance not adding up to my current balance when I check it in my bank app. I had this issue before when initially setting up Actual and syncing the past 3 months, but figured it had to do with historic data and corrected it manually.

What I noticed is that this particular transaction doesn't have a transactionId, it shows: "transactionId":" " in the log, where all the other transactions have some value shown there. Seeing that the transaction shows up in the log (both browser and server) and the current balance also matches, it seems the data is complete, but not shown/processed in Actual. The transaction in question has entryReference 20240320-90950171.

Server log:

Available (first 10) transactions properties for new integration of institution in calculateStartingBalance function {
  balances: '[{"balanceAmount":{"amount":"259.68","currency":"EUR"},"balanceType":"interimAvailable","lastChangeDateTime":"2024-03-23T12:00:00Z"}]',
  top10SortedTransactions: '[{"transactionId":"SCT2024032322412000000024869","entryReference":"20240323-90305441","endToEndId":"NOTPROVIDED","bookingDate":"2024-03-23","valueDate":"2024-03-23","transactionAmount":{"amount":"1.63","currency":"EUR"},"debtorName":"x","debtorAccount":{"iban":"x"},"remittanceInformationUnstructured":"x","bankTransactionCode":"PMNT-RCDT-STDO","proprietaryBankTransactionCode":"FOVS","internalTransactionId":"91d400a0ca5f3cb628050c97ad5056b5","date":"2024-03-23"},{"transactionId":"DOMREC18543170SEQ013248DAT20240319","entryReference":"20240321-91366207","endToEndId":"052008198104","mandateId":"000004880265","creditorId":"NL24ZZZ080534100002","bookingDate":"2024-03-21","valueDate":"2024-03-21","transactionAmount":{"amount":"-1.1","currency":"EUR"},"creditorName":"x","creditorAccount":{"iban":"x"},"remittanceInformationUnstructured":"x","purposeCode":"OTHR","bankTransactionCode":"PMNT-RDDT-ESDD","proprietaryBankTransactionCode":"FEIC","internalTransactionId":"e865f3cf0561559c418fd74612fdc26c","date":"2024-03-21"},{"transactionId":"1033241304979","entryReference":"20240321-91193593","endToEndId":"1033241304979","mandateId":"4LVJ2253GDJXA","creditorId":"LU96ZZZ0000000000000000058","bookingDate":"2024-03-21","valueDate":"2024-03-21","transactionAmount":{"amount":"-108","currency":"EUR"},"creditorName":"x":{"iban":"x"},"remittanceInformationUnstructured":"x","bankTransactionCode":"PMNT-RDDT-ESDD","proprietaryBankTransactionCode":"FEIC","internalTransactionId":"065f218d6ee283515b4b00afe8a55d6d","date":"2024-03-21"},{"transactionId":"SCT2024032022310000000002242","entryReference":"20240320-92134263","endToEndId":"2024-03-20T14:08 1030000851405660","bookingDate":"2024-03-20","valueDate":"2024-03-20","transactionAmount":{"amount":"-1.09","currency":"EUR"},"creditorName":"x","creditorAccount":{"iban":"x"},"remittanceInformationUnstructured":"x","purposeCode":"EPAY","bankTransactionCode":"PMNT-ICDT-ESCT","proprietaryBankTransactionCode":"FIDE","internalTransactionId":"f5690a14aec2886cd5eccb1f7df60e77","date":"2024-03-20"},{"transactionId":" ","entryReference":"20240320-90950171","endToEndId":"NOT_PROVIDED","bookingDate":"2024-03-20","valueDate":"2024-03-20","transactionAmount":{"amount":"34.94","currency":"EUR"},"debtorName":"x","debtorAccount":{"iban":"x"},"remittanceInformationUnstructured":"x","bankTransactionCode":"PMNT-RCDT-OTHR","proprietaryBankTransactionCode":"FBVZ","internalTransactionId":"973003b37f79ef2a95321172c0b420e3","date":"2024-03-20"},{"transactionId":"OO9T003091924877","entryReference":"20240319-91602339","endToEndId":"tra-100696430e66","bookingDate":"2024-03-19","valueDate":"2024-03-19","transactionAmount":{"amount":"20","currency":"EUR"},"debtorName":"x","debtorAccount":{"iban":"x"},"remittanceInformationUnstructured":"x","bankTransactionCode":"PMNT-RRCT-ESCT","proprietaryBankTransactionCode":"FIOS","internalTransactionId":"8dff59650d685907c12fa3386b70451e","date":"2024-03-19"},{"transactionId":"SCT2024031922253000000000126","entryReference":"20240319-91549657","endToEndId":"2024-03-19T15:31 0140000855715664","bookingDate":"2024-03-19","valueDate":"2024-03-19","transactionAmount":{"amount":"-37.44","currency":"EUR"},"creditorName":"x","creditorAccount":{"iban":"x"},"remittanceInformationUnstructured":"x","purposeCode":"EPAY","bankTransactionCode":"PMNT-ICDT-ESCT","proprietaryBankTransactionCode":"FIDE","internalTransactionId":"93539c621690494f32adf4410fc9e7a7","date":"2024-03-19"},{"transactionId":"SCT2024031922143000000010480","entryReference":"20240319-90114341","endToEndId":"NOTPROVIDED","bookingDate":"2024-03-19","valueDate":"2024-03-19","transactionAmount":{"amount":"-150","currency":"EUR"},"creditorName":"x","creditorAccount":{"iban":"x"},"remittanceInformationUnstructured":"x","bankTransactionCode":"PMNT-ICDT-STDO","proprietaryBankTransactionCode":"FPOV","internalTransactionId":"188379ff4169d1e61ab229213e9ddb39","date":"2024-03-19"},{"transactionId":"SCT2024031621991000000010939","entryReference":"20240316-90262435","endToEndId":"NOTPROVIDED","bookingDate":"2024-03-16","valueDate":"2024-03-16","transactionAmount":{"amount":"1.63","currency":"EUR"},"debtorName":"x","debtorAccount":{"iban":"x"},"remittanceInformationUnstructured":"x","bankTransactionCode":"PMNT-RCDT-STDO","proprietaryBankTransactionCode":"FOVS","internalTransactionId":"52f3eb66a4da8240f6412dcbdfc2f5fb","date":"2024-03-16"},{"transactionId":"SCT2024031621991000000019073","entryReference":"20240316-90238929","endToEndId":"NOTPROVIDED","bookingDate":"2024-03-16","valueDate":"2024-03-16","transactionAmount":{"amount":"-48.88","currency":"EUR"},"creditorName":"x","creditorAccount":{"iban":"x"},"remittanceInformationUnstructured":"x","bankTransactionCode":"PMNT-ICDT-STDO","proprietaryBankTransactionCode":"FPOV","internalTransactionId":"ab14b0e3cfac1477e0a4a68ad2282aa0","date":"2024-03-16"}]'
}

Browser log;

{
    "status": "ok",
    "data": {
        "iban": "x",
        "balances": [
            {
                "balanceAmount": {
                    "amount": "259.68",
                    "currency": "EUR"
                },
                "balanceType": "interimAvailable",
                "lastChangeDateTime": "2024-03-23T12:00:00Z"
            }
        ],
        "institutionId": "x",
        "startingBalance": 19305,
        "transactions": {
            "booked": [
                {
                    "transactionId": "SCT2024032322412000000024869",
                    "entryReference": "20240323-90305441",
                    "endToEndId": "NOTPROVIDED",
                    "bookingDate": "2024-03-23",
                    "valueDate": "2024-03-23",
                    "transactionAmount": {
                        "amount": "1.63",
                        "currency": "EUR"
                    },
                    "debtorName": "x",
                    "debtorAccount": {
                        "iban": "x"
                    },
                    "remittanceInformationUnstructured": "x",
                    "bankTransactionCode": "PMNT-RCDT-STDO",
                    "proprietaryBankTransactionCode": "FOVS",
                    "internalTransactionId": "91d400a0ca5f3cb628050c97ad5056b5",
                    "date": "2024-03-23"
                },
                {
                    "transactionId": "DOMREC18543170SEQ013248DAT20240319",
                    "entryReference": "20240321-91366207",
                    "endToEndId": "052008198104",
                    "mandateId": "000004880265",
                    "creditorId": "NL24ZZZ080534100002",
                    "bookingDate": "2024-03-21",
                    "valueDate": "2024-03-21",
                    "transactionAmount": {
                        "amount": "-1.1",
                        "currency": "EUR"
                    },
                    "creditorName": "x",
                    "creditorAccount": {
                        "iban": "x"
                    },
                    "remittanceInformationUnstructured": "x",
                    "purposeCode": "OTHR",
                    "bankTransactionCode": "PMNT-RDDT-ESDD",
                    "proprietaryBankTransactionCode": "FEIC",
                    "internalTransactionId": "e865f3cf0561559c418fd74612fdc26c",
                    "date": "2024-03-21"
                },
                {
                    "transactionId": "1033241304979",
                    "entryReference": "20240321-91193593",
                    "endToEndId": "1033241304979",
                    "mandateId": "4LVJ2253GDJXA",
                    "creditorId": "LU96ZZZ0000000000000000058",
                    "bookingDate": "2024-03-21",
                    "valueDate": "2024-03-21",
                    "transactionAmount": {
                        "amount": "-108",
                        "currency": "EUR"
                    },
                    "creditorName": "x",
                    "creditorAccount": {
                        "iban": "x"
                    },
                    "remittanceInformationUnstructured": "x",
                    "bankTransactionCode": "PMNT-RDDT-ESDD",
                    "proprietaryBankTransactionCode": "FEIC",
                    "internalTransactionId": "065f218d6ee283515b4b00afe8a55d6d",
                    "date": "2024-03-21"
                },
                {
                    "transactionId": "SCT2024032022310000000002242",
                    "entryReference": "20240320-92134263",
                    "endToEndId": "2024-03-20T14:08 1030000851405660",
                    "bookingDate": "2024-03-20",
                    "valueDate": "2024-03-20",
                    "transactionAmount": {
                        "amount": "-1.09",
                        "currency": "EUR"
                    },
                    "creditorName": "x",
                    "creditorAccount": {
                        "iban": "x"
                    },
                    "remittanceInformationUnstructured": "x",
                    "purposeCode": "EPAY",
                    "bankTransactionCode": "PMNT-ICDT-ESCT",
                    "proprietaryBankTransactionCode": "FIDE",
                    "internalTransactionId": "f5690a14aec2886cd5eccb1f7df60e77",
                    "date": "2024-03-20"
                },
                {
                    "transactionId": " ",
                    "entryReference": "20240320-90950171",
                    "endToEndId": "NOT_PROVIDED",
                    "bookingDate": "2024-03-20",
                    "valueDate": "2024-03-20",
                    "transactionAmount": {
                        "amount": "34.94",
                        "currency": "EUR"
                    },
                    "debtorName": "x",
                    "debtorAccount": {
                        "iban": "x"
                    },
                    "remittanceInformationUnstructured": "x",
                    "bankTransactionCode": "PMNT-RCDT-OTHR",
                    "proprietaryBankTransactionCode": "FBVZ",
                    "internalTransactionId": "973003b37f79ef2a95321172c0b420e3",
                    "date": "2024-03-20"
                },

The transaction in question is a 'Payment request'. You can send these using the bank app to other people with a link so they can pay you. In this case it was a friend paying me back for a concert ticket.

I run Actual v24.3.0 as a Home Assistant add-on.

What error did you receive?

None

Where are you hosting Actual?

Other

What browsers are you seeing the problem on?

Chrome

Operating System

Windows 11

👋 First of all: thank you for providing all this data! It's much easier to help if all the necessary logs are provided.

The transaction in question is a 'Payment request'. You can send these using the bank app to other people with a link so they can pay you. In this case it was a friend paying me back for a concert ticket.

Are you sure your friend has made the payment? Has it cleared? I find it strange that you can make a payment request to a 3rd party and that it would affect your ledger prior to actually receiving the funds (i.e. what happens if you make a 10m payment request from your friend? Does that temporarily make your balance go up by 10m?).

After receiving the monies from your friend and the transaction clearing (in a few days from today I guess?) - does the transactionId appear as a non-empty value?

Internally the problem is that " " (transactionId) is actually treated as a valid value. So we look-up your ledger in Actual to find a transaction with the id " ". And I'm guessing you've had previous payment requests, so it is in fact matched. And because it is matched (thus existing in Actual) - we update it and move on instead of creating a new one.

We might need to add some logic here to trim the transactionId values. That way " " would be turned into "" and it would thus NOT be a valid transaction-id to use for matching. And thus it would try to match by date & transaction amount instead of transactionId.

But this is dangerous as it is a breaking change that can impact other users of Actual. Would need to be careful.

👋 First of all: thank you for providing all this data! It's much easier to help if all the necessary logs are provided.

The transaction in question is a 'Payment request'. You can send these using the bank app to other people with a link so they can pay you. In this case it was a friend paying me back for a concert ticket.

Are you sure your friend has made the payment? Has it cleared? I find it strange that you can make a payment request to a 3rd party and that it would affect your ledger prior to actually receiving the funds (i.e. what happens if you make a 10m payment request from your friend? Does that temporarily make your balance go up by 10m?).

After receiving the monies from your friend and the transaction clearing (in a few days from today I guess?) - does the transactionId appear as a non-empty value?

--

Internally the problem is that " " (transactionId) is actually treated as a valid value. So we look-up your ledger in Actual to find a transaction with the id " ". And I'm guessing you've had previous payment requests, so it is in fact matched. And because it is matched (thus existing in Actual) - we update it and move on instead of creating a new one.

We might need to add some logic here to trim the transactionId values. That way " " would be turned into "" and it would thus NOT be a valid transaction-id to use for matching. And thus it would try to match by date & transaction amount instead of transactionId.

But this is dangerous as it is a breaking change that can impact other users of Actual. Would need to be careful.

Thanks for the explanation. To answer your questions: the payment request is created in my bank app, and only shows up in the transactions once someone actually pays. The payments are always instantly (all Dutch banks support SEPA instant) received. So the transaction I tried to sync is the final transaction, there is no clearing involved as it's always instant and only shows up (becomes a transaction) when someone pays.

What's the institution ID for this bank? We could create a custom bank mapper for it to trim the transactionId values. I think that's the safest solution here.

What's the institution ID for this bank? We could create a custom bank mapper for it to trim the transactionId values. I think that's the safest solution here.

The institution ID is ASN_BANK_ASNBNL21. Thanks!

👋 I was chatting with @kyrias and he noted that according to GoCardless documentation the transactionId field has to be unique (and optional; source). In your case it does not seem to be unique.

Which leads us to believe the patch should actually be on GoCardless side. Could you reach out to their support and see if they can patch it on their side?

Thanks

👍 I've send in a ticket to support about this. I noticed that basically all the 'internal' transactions (paid payment requests, account fees, interest payments) have transaction ID " ".

When I send in the ticket I saw this support article. Why does Actual not use the internalTransactionID? It seems that one is truly unique and already filtered by GoCardless for duplicates.

Just got a response from GoCardless. They suggest to use the internalTransactionID;

Because it didn't exist when the GoCardless (or then Nordigen) support was added, and switching at this point would need to be done very carefully to not break existing users.

The options I can think of are:

Add a metadata field to the account that's set the first time the account is linked (and have a migration that sets it on all existing accounts) that decides whether to use transactionId or internalTransactionId. This seems like it would probably be the simplest option, but it would mean that we need to continue supporting using both IDs essentially forever, and it means adding GoCardless-specific metadata to the account which is only needed for this one little thing.
Make the server return both and support using both for matching pre-existing transactions while saving new ones using the internal one. This would be a bit more work, but I think it's the more future-proof option since it means that we could explicitly say that we're going to drop support for the old IDs after a couple of releases and that people need to ensure that they've upgraded and synced at least ~30 days before upgrading to the release that drops support for the old IDs. Though doing this I think we should probably prepend some metadata to the new ID to prevent both accidental conflicts between old transaction IDs and new GoCardless-assigned ones, and conflicts with internal transaction IDs between accounts since they're only guaranteed to be unique within a given account. Something like gocardless-{resourceId}-{internalTransactionId} maybe?
Make imported_id an array, and always use both when they're available. This just seems like it'd make the application more complicated for no real advantage.

Also, I find it so annoying how GoCardless support responses to their API not working the way it's documented is always for you to implement random work-arounds rather than them cleaning up the data to correspond to their own documented schema.

+1 to @kyrias

Ideally we should push more on GoCardless here to either patch it on their end or update the documentation to reflect the reality (transactionId NOT being unique).

I will also leave this issue open in case anyone wants to pick up the refactors to gracefully switch over to internalTransactionId.

Just as an observation from my Nationwide account I sync via GoCardless, it doesn't look like internalTransactionId is populated until the transaction is booked.

Pending

            {
                "transactionId": "xxxx",
                "bookingDate": "2024-04-19",
                "bookingDateTime": "2024-04-29T12:31:29Z",
                "transactionAmount": {
                    "amount": "-3.75",
                    "currency": "GBP"
                },
                "creditorName": "xxxx",
                "remittanceInformationUnstructured": "xxxx"
            }

Booked

            {
                "transactionId": "xxxx",
                "bookingDate": "2024-04-18",
                "bookingDateTime": "2024-04-18T00:00:00Z",
                "transactionAmount": {
                    "amount": "-3.75",
                    "currency": "GBP"
                },
                "creditorName": "xxxx",
                "remittanceInformationUnstructured": "xxxx",
                "internalTransactionId": "xxxx"
            },

The GoCardless docs don't seem to mention this at all, and I'm unsure whether it's inconsistent between different banks.

Hey guys is there any update on the issue. I'm experiencing very similar problem where some transactions do not have unique ID. In my case the field is not empty but it is the same as several other transactions. Interestingly the affected transactions seem to be the ones where I sent manually, It has never happened for card payments.

Anyway thanks for all the hard work, Actual is pretty great.

actualbudget / actual