Data importer gives several different errors

krispek commented 1 year ago

Support guidelines

[X] I've read the support guidelines

I've found a bug and checked that ...

[X] ... the documentation does not mention anything about my problem
[X] ... there are no open or closed issues that are related to my problem
[X] ... it's definitely a Firefly III issue, not me

Description

Not sure whether this is a follow up of #7871 or #7536. Firefly gives several issues when importing transactions

Issue 1 is a transfer:

[2023-08-25 10:09:37] production.DEBUG: Duplicate detection method is "classic", so this method is skipped (return true).  
[2023-08-25 10:09:37] production.DEBUG: Transaction #41 is not checked beforehand on uniqueness.  
[2023-08-25 10:09:37] production.DEBUG: Going to map data for this line.  
[2023-08-25 10:09:37] production.DEBUG: Submitting to Firefly III: {"apply_rules":true,"error_if_duplicate_hash":true,"transactions":[{"type":"withdrawal","date":"2023-08-21 00:00:00","amount":"979.000000000000","description":"N/A","order":0,"currency_code":"EUR","external_id":"1085583244188981941","interal_reference":"1085583192422881644","notes":"PAYMENT","source_id":1145,"destination_name":"my name","destination_iban":"NL11BUNQ11111111111"}]} 
[2023-08-25 10:09:37] production.ERROR: Submission error: 0 ["De IBAN van deze doelrekening is al in gebruik door een betaalrekening of een passiva, en kan niet worden gebruikt als een doelrekening voor deze uitgave."] 
[2023-08-25 10:09:37] production.DEBUG: This is not a duplicate transaction error  
[2023-08-25 10:09:37] production.DEBUG: Add error on index #40 (line no. 41): transactions.0.destination_id: De IBAN van deze doelrekening is al in gebruik door een betaalrekening of een passiva, en kan niet worden gebruikt als een doelrekening voor deze uitgave. (original value: "(not found)")  
[2023-08-25 10:09:37] production.DEBUG: Now in App\Services\Shared\Import\Status\SubmissionStatusManager::storeSubmissionStatus(conv-6LkknNudph65): submission_running  
[2023-08-25 10:09:37] production.ERROR: transactions.0.destination_id: De IBAN van deze doelrekening is al in gebruik door een betaalrekening of een passiva, en kan niet worden gebruikt als een doelrekening voor deze uitgave. (original value: "(not found)")  
[2023-08-25 10:09:37] production.ERROR: Submission error: 0 ["De IBAN van deze doelrekening is al in gebruik door een betaalrekening of een passiva, en kan niet worden gebruikt als een doelrekening voor deze uitgave."] 
[2023-08-25 10:09:37] production.DEBUG: This is not a duplicate transaction error  
[2023-08-25 10:09:37] production.DEBUG: Add error on index #40 (line no. 41): transactions.0.destination_name: De IBAN van deze doelrekening is al in gebruik door een betaalrekening of een passiva, en kan niet worden gebruikt als een doelrekening voor deze uitgave. (original value: "my name")  
[2023-08-25 10:09:37] production.DEBUG: Now in App\Services\Shared\Import\Status\SubmissionStatusManager::storeSubmissionStatus(conv-6LkknNudph65): submission_running  
[2023-08-25 10:09:37] production.ERROR: transactions.0.destination_name: De IBAN van deze doelrekening is al in gebruik door een betaalrekening of een passiva, en kan niet worden gebruikt als een doelrekening voor deze uitgave. (original value: "my name")  
[2023-08-25 10:09:37] production.DEBUG: Group is empty, may not have been stored correctly.

Next is a incoming transaction:

[2023-08-25 10:31:35] production.DEBUG: Duplicate detection method is "classic", so this method is skipped (return true).  
[2023-08-25 10:31:35] production.DEBUG: Transaction #42 is not checked beforehand on uniqueness.  
[2023-08-25 10:31:35] production.DEBUG: Going to map data for this line.  
[2023-08-25 10:31:35] production.DEBUG: Submitting to Firefly III: {"apply_rules":true,"error_if_duplicate_hash":true,"transactions":[{"type":"deposit","date":"2023-08-21 00:00:00","amount":"979","description":"description","order":0,"currency_code":"EUR","external_id":"1085583244188981942","interal_reference":"1085583192422881644","notes":"SCT","destination_id":1145,"source_name":"my name","source_iban":"NL11BUNQ11111111"}]} 
[2023-08-25 10:31:35] production.ERROR: Submission error: 0 ["No error yet."] 
[2023-08-25 10:31:35] production.DEBUG: This is not a duplicate transaction error  
[2023-08-25 10:31:35] production.DEBUG: Add error on index #41 (line no. 42): transactions.0.source_id: No error yet. (original value: "(not found)")  
[2023-08-25 10:31:35] production.DEBUG: Now in App\Services\Shared\Import\Status\SubmissionStatusManager::storeSubmissionStatus(conv-ePvopuWj6POE): submission_running  
[2023-08-25 10:31:35] production.ERROR: transactions.0.source_id: No error yet. (original value: "(not found)")  
[2023-08-25 10:31:35] production.ERROR: Submission error: 0 ["No error yet."] 
[2023-08-25 10:31:35] production.DEBUG: This is not a duplicate transaction error  
[2023-08-25 10:31:35] production.DEBUG: Add error on index #41 (line no. 42): transactions.0.source_name: No error yet. (original value: "my name")  
[2023-08-25 10:31:35] production.DEBUG: Now in App\Services\Shared\Import\Status\SubmissionStatusManager::storeSubmissionStatus(conv-ePvopuWj6POE): submission_running  
[2023-08-25 10:31:35] production.ERROR: transactions.0.source_name: No error yet. (original value: "my name")  
[2023-08-25 10:31:35] production.DEBUG: Group is empty, may not have been stored correctly.

Next, for card transactions, I get the following:

[2023-08-25 10:16:31] production.DEBUG: Payee is "", payee info is "CCV*companyname", return "unknown".  
[2023-08-25 10:16:31] production.DEBUG: isValidIBAN("")  
[2023-08-25 10:16:31] production.DEBUG: Trim: isValidIBAN("")  
[2023-08-25 10:16:31] production.DEBUG: isValidIBAN("CCV* companyname")  
[2023-08-25 10:16:31] production.DEBUG: Trim: isValidIBAN("CCV* companyname")  
[2023-08-25 10:16:31] production.ERROR: Bad IBAN: bcmod(): Argument #1 ($num1) is not well-formed  
[2023-08-25 10:16:31] production.DEBUG: Payee IBAN is "" (empty fallback)  
[2023-08-25 10:16:31] production.DEBUG: source_id = 1145, destination_name = "(unknown destination account)", destination_iban = ""

The result over here is that the transaction is stored under "unknown destination account" not under "CCV*companyname"

Debug information

Debug information generated at 2023-08-25 10:47:08 Europe/Berlin for Firefly III Data Importer version 1.3.5.

Scope	Version
FIDI	1.3.5
Build	281, 22-08-2023 04:28:11 UTC
PHP	8.2.8
Host	Linux

System info	Value
Using docker?	true
System TZ	Europe/Berlin
Browser TZ	Europe/Amsterdam
App environment	production
App debug mode	false
App cache driver	file
App logging	debug, stack
Display errors	Off
Error reporting	ALL errors
Interface	apache2handler
BCscale	12
Trusted proxies (.env)

User info	Value
User agent	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36

Expected behaviour

Transactions are stored The right destination name is used for a transaction

Steps to reproduce

Use spectre
Choose bunq
Import some transactions

Additional info

No response

JC5 commented 1 year ago

Could you share (in private) the logs of the data importer before it reaches the errors?

BWFCGUIXT commented 1 year ago

This is such a coincidence, I was coming here to post the exact same thing!

BWFCGUIXT commented 1 year ago

I also noticed only the Expenses are being imported, the Income is not getting imported anymore.

Some of the errors:

Line #49: (2)transactions.0.destination_id: This destination account IBAN is already in use by an asset account or a liability and cannot be used as a withdrawal destination. (original value: "(not found)")

Line #7: (2)transactions.0.source_id: No error yet. (original value: "(not found)") (2)transactions.0.source_name: No error yet. (original value: "NAME")

JC5 commented 1 year ago

Could also send me some logs?

BWFCGUIXT commented 1 year ago

Member

Sure is this the file you need: ff3-cli-2023-08-25.log / ff3-audit-2023-08-25.log

Also seeing this in my docker logs: (The Bold one is in red, and has me worried I messed up something bad)

[!] User #1 (MY USER NAME) has account #1508 ("IBAN") which has no transactions. [!] User #1 (MY USER NAME) has account #1509 ("IBAN") which has no transactions. [!] User #1 (MY USER NAME) has account #1522 ("IBAN") which has no transactions. [!] User #1 (MY USER NAME) has account #1527 ("IBAN") which has no transactions. [!] User #1 (MY USER NAME) has account #1554 ("IBAN") which has no transactions. Now executing firefly-iii:report-sum [x] Error: Transactions for user #1 (MY USER NAME) are off by 0.000000000047363327562477!

JC5 commented 1 year ago

I'll need the logs from the data importer.

To fix that particular error, make a backup of your database and then run this command.

php artisan firefly-iii:force-decimal-size

BWFCGUIXT commented 1 year ago

Thanks for the help on the error 👍🏻

I have the data importer running in docker, how can I obtain the log files for you?

I can run an import today, and send you the log.

JC5 commented 1 year ago

:+1: docker logs -f containername

BWFCGUIXT commented 1 year ago

Hi James,

Send you an email with the log 👍🏻

BWFCGUIXT commented 1 year ago

Thanks for the command :) the error is gone now!

Should I also do something about this warning or it's fine?

[!] User https://github.com/firefly-iii/firefly-iii/issues/1 (MY USER NAME) has account https://github.com/firefly-iii/firefly-iii/issues/1554 ("IBAN") which has no transactions.

JC5 commented 1 year ago

Well the text explains exactly what is happening. You don't need to do anything about it :)

JC5 commented 1 year ago

If you could both try the develop tag of the data importer, I have pushed a potential fix.

If this doesn't seem to fix the issue, please send me the logs of both the data importer and Firefly III, preferably from around the same time (so I can compare notes).

BWFCGUIXT commented 1 year ago

I went out and bought something :)

Tried develop but the issue remains, emailed you the log.

JC5 commented 1 year ago

Nice, I think I found it. If you could try pulling develop again? It should report as build 287

BWFCGUIXT commented 1 year ago

Done and indeed it works 👍 my expenses are showing the destination account again.

But Revenue / Income transactions still aren't imported, so something must be wrong on that side.

JC5 commented 1 year ago

Same routine basically. Run into an error, send me the logs! 👍

BWFCGUIXT commented 1 year ago

Logs are on the way :)

Was expecting a Paypal incoming transaction.

PAYPAL | €23.80 | August 25th, 2023

I did add it manually already, not sure if that is conflicting with the importer.

JC5 commented 1 year ago

@krispek, I suspect many errors have been caught already, the logs supplied by @BWFCGUIXT have been very helpful. Will you let me know what you run into when you use the develop branch of the data importer?

krispek commented 1 year ago

@JC5 I sent you a mail. :) Was some time off. I tested with build 294. Thanks for investigating!

JC5 commented 1 year ago

Fixed in the next data importer release.

github-actions[bot] commented 1 year ago

Hi there! This is an automatic reply. Share and enjoy

This issue is now 🔒 closed. Please be aware that closed issues are not watched.

If the original bug is not actually fixed, please feel free to open a new issue. Please refer to this issue for clarity.
Follow-up questions must be posted in a new discussion
Further replies to this issue will get no response.

Thank you for your contributions.

firefly-iii / firefly-iii