[fireflyiii_data_importer] Option to run automatically command line on addon startup/repeat

[diamant-x]

Which addon? Firefly iii Data Importer

Is your feature request related to a problem? Please describe. The Data Importer runs fine but currently requires manual action to import it, which is not useful to automate and have a low effort personal finance tracker. The source docker program allows for imports to be performed through CLI too, and the author recommend setting up a cron or something like that to trigger the command line to auto-import.

Describe the solution you'd like The addon would feature a new config boolean askign whether the addon should attempt autoimport on startup. Then also the addon would look at a specific folder under its config and proceed to load what files it finds as per the described documentation here.

Describe alternatives you've considered The only alternative I can think of is having a keyboard and mouse recorded to do this steps, but even that wouldn't allow for an unattended server setup.

Additional context According to the docs, fors this to work it should be implemented that: if the config flag is marked, then the docker should run at the end of its startup the line:

# docker
docker exec -it <container> php artisan importer:auto-import /import
#self hosted
php artisan importer:auto-import /path/to/your/files

where /import is the config folder with the files to import.

[alexbelgium]

Thanks for the detailed and precise request. I'll look at it as soon as possible

[diamant-x]

Thanks for the prompt feedback! Looking forward to your findings.

I see there's related .env variables available, although not sure if them alone would be enough for this to already be working "as is" or still some modifications would be needed on the dockerfile.

#
# Auto import settings. Due to security constraints, you MUST enable each feature individually.
# You must also set a secret. The secret is used for the web routes.
#
# The auto-import secret must be a string of at least 16 characters.
# Visit this page for inspiration: https://www.random.org/passwords/?num=1&len=16&format=html&rnd=new
#
# Submit it using ?secret=X
#
AUTO_IMPORT_SECRET=

#
# Is the /autoimport even endpoint enabled?
# By default it's disabled, and the secret alone will not enable it.
#
CAN_POST_AUTOIMPORT=false

#
# Is the /autoupload endpoint enabled?
# By default it's disabled, and the secret alone will not enable it.
#
CAN_POST_FILES=false

#
# Import directory white list. You need to set this before the auto importer will accept a directory to import from.
#
IMPORT_DIR_WHITELIST=

[alexbelgium]

Hello, so I've enabled cron, for which you can select update frequency as hourly/daily/weekly. It runs an equivalent to php artisan importer:auto-import /path/to/your/files ; where /path/to/your/files corresponds to the CONFIG_LOCATION/import_configs ; CONFIG_LOCATION being described in the addon options as default /config/addons_config/fireflyiii_data_importer. So, the location where files should be added is /config/addons_config/fireflyiii_data_importer/import_files.

Hope it will work :-) I don't have a config so if you have some time let's troubleshoot that together - if you can run it and tell me if you have any error messages! Thanks!

(if what I noted makes no sense - just run the addon and everything will be in the log :-) )

[diamant-x]

Hi, Thanks for the quick implementation. I looked at the new folder structure (within CONFIG_LOCATION) and saw that there's now 2 folders, and the naming has a trailing close parenthesis sign. Could you check that maybe remove the extra ")" and identify the difference between these 2 folders?

Also, the configuration pane now has two 'empty' entries. i understand one would be to disable the periodic cron, but the other one?

Thanks.

[alexbelgium]

Hi, thanks for the feedback. I'll correct those in the version I've pushed now. The two empty configs options were the same, they just resulted from a setting that I've not disabled.

Here is the difference between both (I've added it in the readme) :

• Configurations can be added in the /config/addons_config/fireflyiii_data_importer/configurations folder according to :https://docs.firefly-iii.org/data-importer/help/config/
• An auto import can be made by adding files in /config/addons_config/fireflyiii_data_importer/import_files according to : https://docs.firefly-iii.org/data-importer/usage/command_line/

[diamant-x]

Hi, Sorry busy during the workweek. I had this running during the week without any import file just to check if the addon failed. Seems to be stable at least so that's good news. Then on the auto-import, the files are recognised by the tool on startup. It loads the configs fine in the dropdown options. For the autoimport it identifies the selected schedule, but i'm not sure it is working through cron at all. I had it running since 09am aprox, with an hourly autoimport, and then updated the file to import at 18h, but at 20.47h it still hasn't been imported. Logs don't show anything either (not sure if it is programmed to output the autoimport logs or not). "Silent" config is not selected.

[09:03:18] INFO: hourly updates
Starting periodic command scheduler: cron.
[09:03:18] INFO: Automatic updates were requested. The files in /config/addons_config/fireflyiii_data_importer/import_files will be imported hourly.
[09:03:18] INFO: Please wait while the app is loading !
Now in entrypoint.sh for Firefly III Data Importer
Script:        1.5 (2021-12-05)
User:          'root'
Group:         'root'
Working dir:   '/var/www/html'
Build number:  93
Build date:    06-02-2022 09:21:56 CET
Firefly III data importer v0.8.0
PHP: cli 8.0.13 Linux
Will now run Apache web server:
[Fri Feb 11 09:03:27.621307 2022] [mpm_prefork:notice] [pid 296] AH00163: Apache/2.4.38 (Debian) configured -- resuming normal operations
[Fri Feb 11 09:03:27.622549 2022] [core:notice] [pid 296] AH00094: Command line: 'apache2 -D FOREGROUND'

[diamant-x]

More tests. I tried restarting the addon to see if at least on startup it would import the files but it didn't either. The logs show no change either so I can't tell if it didn't run at all, or if it run but the import failed.

[alexbelgium]

Thanks, my pc broke down but I'll look when it will be possible again. This means the cron doesn't run, as it's first action is anyway a log message saying it is running... My guess is that I used cron.hourly but that it is not implemented in the base distribution, I'll investigate further

BTW, thanks very much for the very detailed infos. Very helpful for me to pinpoint the issue

[alexbelgium]

Ah... So it seems cron run but it's output is not redirected to docker logs. I'll do some tests

[alexbelgium]

Hi, now it's done :-) I've redirected the messages from cron to > /proc/1/fd/1 which allows them to appear in supervisor. At least it works from portainer, I'll now update the addon and see if it works as expected

[alexbelgium]

I'll check if it works and report here

[alexbelgium]

typos :-) version-0.8.0-4 will work :-)

[alexbelgium]

(version-0.8.0-4: well I can't fully test as I don't have a running Firefly instance ; but cron jobs executes and shows its output in logs)

[alexbelgium]

I've pushed a new version, should be good : Firefly version-0.8.0-4 ; Data Importer 5.6.14-3

[diamant-x]

Great. I can see the logs in the Portainer, but not in the Addon logs though. At least I can see that I have an issue in my import files. I'll try to fix it along this week and report back if everything works fine. Thanks!! [2022-02-14 20:17:03] production.DEBUG: Add error on index #0 (line no. 1): Could not parse CSV: Could not decode JSON line #2: Malformed UTF-8 characters, possibly incorrectly encoded

[diamant-x]

Ok, I updated my import .csv file and it was picked up correctly now (It had to be exported as UTF_8). It was now picked up, and the add-on logs show the full list of messages from the autoimporter action. It would say this feature is now working as expected :-).

• Just one last question, would the cron trigger on add-on start, or it needs to wait until the first recurrence occurrence (i.e., if set to hourly, the first attempt will be done after 1h since Addon start)?

Thanks for the effort! :D

[alexbelgium]

Great news! The cron job works x time after start according to the set duration. But I think it still loads it at boot, just independently from the cron