Customize backup file name and interval

[JeanMiK]

Feature

I am running two different instances of dsmr-reader to read my two Smarty readers (one for house electricity and gas consumption, one for PV production) and want to do Dropbox backups for them into a single Dropbox account.

Initially, I was using only a free Dropbox account, and was therefore facing some space issues, which lead me to raise the issue #1452 (requesting the possibility to set a specific fix name of the backup files, so that Dropbox itself would do the versioning of the backups, and the whole set of backups only being 'counted' with the size of the most recent backup).

As there was no plan to integrate any such change, I decided at the time to simply go for a 'real', 'for money' Dropbox account, which solved the space issue, and all was well with the Dsmrreader versions 4.x.

Now, however, I have finally switched over to the 5.x Dsmrreader version, and I encounter a new problem: before, I was using a different 'application' in Dropbox for each of the two instances, so that the two sets of backups would automatically go to two different places in the account. Now, however, if I would activate backups on both of the instances, they would both be writing files of the same name in the "Official DSMR-Reader", so this would not be working.

It would be good to have a way to enable backups from two different instances to the same Dropbox account even with the 'Official DSMR-Reader' Dropbox registration.

This could be done e.g. by allowing in the adminstrative GUI to enter a sub-directory in the 'Official DSMR-Reader' and then to prefix all filepaths with this sub-directory

[dennissiemensma]

I will have a look at it for the next release or the one after.

Eventually I will remove the built-in Dropbox-support and defer it to a separate Docker container, allowing other Dropbox-like solutions to be used as well. Which has been a wish for some users for a long time, but should not be a part of DSMR-reader anyway.

However, the basic backup setup will probably stay in order to allow different preferences. So configuring the interval and name will fit better in such future setup.

[dennissiemensma]

Well it seems you can already prefix the backup file name. See https://github.com/dsmrreader/dsmr-reader/issues/1609#issuecomment-1086293103

I'll still move it to the interface, along with the interval override in #1685.

[JeanMiK]

Excellent, for my needs, the environment variable is already good enough, I can set it directly in the docker-compose file. For other people, it is probably nicer if they can set it in the configuration interface.

Will the enviornment variable still work then, or will I have to switch?

[dennissiemensma]

The env vars are a "temporary" solution, depending on how often you update. They're likely to be dropped in the next major release.

Any changes should always be mentioned in the changelog though, even these undocumented env vars: v5.9 stub

[dennissiemensma]

Additionally, I've reconsidered my take above: https://github.com/dsmrreader/dsmr-reader/issues/1746#issuecomment-1322499749

In the long run only Docker containers will be supported anyway for this project and it's very likely that the backup mechanism will be dropped entirely from this project. It's currently an alias for a pg_dump run anyway, which should be run from whatever database server or database container is used. The main reason being incompatible postgresql-client versions required in a base DSMR-reader container. It's quite easy to run such command from any system anyway, also allowing the end-user to choose whatever interval and filename for the backup. Making it completely obsolete.

However, this surely is not something to change on short term. Bumping the project to containers-only is a big enough first step.

[JeanMiK]

Hmm, I am not sure if the change of the prefix really works for the current Dropbox backup.

After setting the prefix in the docker-compose, I did not see any new back-up files with the new prefix appearing in the Dropbox App folder. After letting it run for a few day without success, I wondered whether there was maybe a typo in my variable name, so I accessed the docker diirectly, and at fiirst, did not find any newer backup file at all in the volume I mounted for the backups.

This, however, turned out to be rather an issue with the dsmr docker container, where now inside the container, the backup directory is under '/app/', so in '/app/backups', where before, it was under '/dsmr', i.e. in '/dsmr/backups'. In the new location, there was indeed a backup file generated with the new prefix.

I fixed the path of my docker volume, and all is well, except that the files wiith the new prefix don't seem to get copied to the drop-box folder.

Am I missing something here, or is file upload maybe only looking at the original file name without taking into account the prefix?

[dennissiemensma]

You can try to enable debug logging, reschedule the Dropbox sync in the DSMR-reader admin scheduled processes to force it and have a look at the dsmr-backend logfiles to see what happens.

[dennissiemensma]

Part of next v5.9 release