DietPi-Software | Preserve or backup existing config files on (re)installs

thaihugo commented 5 years ago

Hi, The latest DietPi date (6.17) completely replaced my /etc/mpd.conf without any backup. I now have to recreate all my entries from scratch to deal with my Bluetooth audio, http streaming and recording plugin. I understand that there are still bugs in the mpd/mympd couple and that I will probably have another deletion down the line on the next update.

I know the official answer for this topic is « do a backup » with a tool that does not allow me to exclude big data folders, or to create my own solution. However, I see this is a trend on DietPi to consider user configuration files as part of the system, and I don’t know where I can go to protect each version from modification.

The previous example was .bashrc and .local. This two files have been hacked so much by updates, that I had to modify them every two weeks as you would delete any line with « DietPi » in it in your process to migrating the banner display process.

My point is that I have to deal with Debian clearly documented configuration rules, then with DietPi not so obvious configuration rules and then with your update process that is clearly the « catch of the day » where I spend some time reading the update log trying to find where it will modify my files and plan for it and then see it will do something unexpected.

When you change daemon users, folder access right, you suppose the user never touch it’s own config. I cannot file a report for every small bug like a chown missing in the Music folder but if I don’t, next update will change the user and I have to re change my solution. This can be quite frustrating. To give you a real example, when you added mympd, the deamon configuration file had a typo and wasn’t launching, the Music folder was owned by DietPi user so mpd could’t access it, and mympd was using mympd user so couldn’t save playlist. I cannot wait one month for an update, so I’ll correct it myself, and next DietPi-update, I’ll have to reassess how you changed files and users to correct the issue and update the new configuration.

Describe the solution you'd like:

Is it possible to create a backup file from each file you are modifying in the update process, so we can go back to them if necessary, and maybe create a local system change log (files modified, deleted, users modified, chown, etc) we can refer to to find why something is not working after an update.

Also, giving more option for user to deal with config outside of DietPi, like for example removing a software from DietPi view but leaving it to be maintained by the user would be nice.

Thank you for your time on all this.

Hugo

Fourdee commented 5 years ago

@thaihugo

Hi Hugo,

Thanks for the report and apologies for the issues caused.

With DietPi, the whole purpose of our applications is to remove the need to modify Linux files, outside of DietPi scripts. However, as per the update prompt, if you modify Linux files, outside of DietPi scripts, they may be modified if a patch is required to bring the system up to next release.

As for which files are modified, you would need to check the sourcecode as its not fee-sable for us to document all changes on file level with our limited dev team (2).

However, I see this is a trend on DietPi to consider user configuration files as part of the system

Exactly this, if we do not apply updates to match all systems and possible installed software across 30K+ installations, DietPi could not exist.

We also added a menu that allows you to backup your system, before selecting the update option. This will appear once v6.16 is installed and next update run.

Is it possible to create a backup file from each file you are modifying in the update process, so we can go back to them if necessary, and maybe create a local system change log (files modified, deleted, users modified, chown, etc) we can refer to to find why something is not working after an update.

Not really fee-sable for us to make this possible. As Linux software is always evolving, and, our small dev team (2), it would be impossible for us to fully maintain this currently, on a level that you could "rely" on.

and maybe create a local system change log

The last update log is available here:

/var/tmp/dietpi/logs/dietpi-update.log

My only suggestion, is as per this window:

If you are modifying the system outside of DietPi scripts, create a new system backup, before updating:

@thaihugo

I understand where you are coming from, and, I wish I could hire 20 people to make the changes needed for us to exclude or check and manually patch each line of the user configuration outside of system updates, document all file changes. However, its just not a possibility for us at the moment.

I can only suggest, that if you are applying manual changes to configurations, that add features, please create a Git ticket with the feature request, and if we implement, we can then fully support it.

MichaIng commented 5 years ago

@thaihugo @Fourdee But I agree, that we should preserve existing configurations/installs as good as we can. In many cases also in-UI configurations are stored in the settings file and/or it is simply required to do manual edit (also according to our own docs in some cases).

I had already in mind to go through all our installs and see where we can either:

Use G_CONFIG_INJECT to place required settings instead of using our own complete config file. This would also reduce the need to maintain code, if e.g. settings or defaults change with software updates.
Use PRESERVE=1 G_CONFIG_INJECT in cases where er recommend but not require a certain value. This is how we add memory cache settings for ownCloud/Nextcloud e.g., so if user actively set up a certain memcache module, we respect this. But this only works if no default value exists on fresh install for the setting.
If the needed adjustments are really too much or complicated to do all of this via G_CONFIG_INJECT, sed etc. in-place, then as a first install step, check for existing config file and do a backup with user notification. This is also what I recently implemented into ownCloud/Nextcloud, before touching the related webserver sites/configs:
```
if [[ -f $owncloud_conf ]]; then

owncloud_conf+='.dietpi-new'
G_WHIP_MSG "Existing ownCloud Apache configuration found, will preserve the old one and save the new one for review and comparison to: $owncloud_conf"
```

fi G_RUN_CMD cp -f /DietPi/dietpi/conf/apache.ownnextcloud.conf $owncloud_conf


This is just an example where I really wanted to preserve any existing config, as if nextcloud.conf exists, user clearly set up and had this working already. In other cases we might want to backup the existing config instead (e.g. to config.dietpi-bak) to assure our new is placed. Of course depends.

So generally I would agree this needs some review and rework in cases. Especially I found a reinstall of webservers to overwrite HTTPS and other changes made (by our other software installs) changes, which will in case break everything, especially if HSTS was active and clients deny to access via HTTP then.
____________
I turn this issue into a feature request/enhancement and will do a start the next days.

thaihugo commented 5 years ago

@Fourdee

Thanks for your explanation and I understand perfectly the need for more ressources. A few point I would like to answer:

The backup function is useful, but if my dietpi_userdata is very big with music and files, I get this in the backup as well. Or let’s say I have a few gigs of docker images, depending on the config, it might go in as well even if I don’t want it saved this way (ie with all incremental file modifications « a la » docker). I wish there was a lower level with the « core » configuration saved as otherwise, I would need double the file system space saved before each update.
The log you referred is too high level to refer to individual file modification. I know it updated mpd, but I don’t know what.
The file backup solution I was talking about is more in the line of a G_command that @MichaIng is explaining very well in his answer. It would be part of the update process the same way apt works asking to keep or modify existing config
The need to be able to modify config by DietPi as a core function is understood by me. But in the case of mpd, we have a grey area: you cannot expect user to only use the base configuration with one ALSA output and that’s it, as it is a server program and usability is in it’s configuration; also you are not getting the packet from raspbian repository and using apt conflict management but un-tarring your own source probably. There is also a /DietPi/DietPi/config/mpd.conf that I suspect is used to reset config from time to time. This is clearly a case where we would need a safe haven for user config, as it is expected to change. It cannot be homefolder/mpd.conf as you sometimes change the deamon user from mpd to DietPi to root for example.

@MichaIng That would be a great solution and what I’m talking about. Thanks for your work.

And thanks both of you for your help and the quality of your answers

Hugo

MichaIng commented 5 years ago

PR in work: https://github.com/Fourdee/DietPi/pull/2214

ID	Title	Configs	Internal updater	Install method
147	Jackett	.config/.mono	🈺 check only	Backup configs, purge old install dir, merge configs into new install
154	RoonServer	?	🈯️ Internal auto updater	Skip, if existent
54	phpBB	config.php/custom extensions	🈯️ Internal update method provided	Skip, if existent
121	RoonBridge	outside of install dir	🈯️ Internal auto updater	Remove existing install dir before installing the new one. This is already current practice 😉.
48	Pydio	`data/` + `conf/`?	🈯️ Internal updater	Skip, if existent
64	PHPSysInfo	/var/www/phpsysinfo/phpsysinfo.ini	?	G_BACKUP_FP /var/www/phpsysinfo/phpsysinfo.ini

... more to come

Fourdee commented 5 years ago

Mmmm, new global for backing up dirs/files?

G_BACKUP_FP

if file/folder, cp to .bak

eg:

G_BACKUP_FP /etc/mpd.conf
G_BACKUP_FP /etc/backthisup

cp + G_DIETPI-NOTIFY 2 ' Backed up' becomes

/etc/mpd.conf.bak
/etc/backthisup.bak

.bak_DDMMYYYY?

🈯️

root@DietPi:~# G_BACKUP_FP /root /tmp
[  OK  ] /root: backup to /root.bak_101118_0
[  OK  ] /tmp: backup to /tmp.bak_101118_0

root@DietPi:~# G_BACKUP_FP /root /tmp
[  OK  ] /root: backup to /root.bak_101118_1
[  OK  ] /tmp: backup to /tmp.bak_101118_1

Fourdee commented 5 years ago

@thaihugo @MichaIng

Ok, added a new feature which backs up configs, before we overwrite during reinstalls:

https://github.com/Fourdee/DietPi/issues/2187#issuecomment-437589994

MPD and a few others are now enabled. We will roll this out to more applications in the future as needed.

I believe we can mark this as completed?

thaihugo commented 5 years ago

@Fourdee yes, the proposed implementation covers my request.

I understand it will provide a mechanism to backup selected files chosen by maintainer and provide a name of saved files in the logs. Any missing file should then become a simple github request to modify the installation script.

Thanks for your work on this. I think this is one of the “under the hood” change that was needed as DietPi grows more versatile over time.

Fourdee commented 5 years ago

@thaihugo

Thanks for your work on this. I think this is one of the “under the hood” change that was needed as DietPi grows more versatile over time.

Yep agree 👍

G_BACKUP_FP is a simple unique backup of configs, before we overwrite during install/reinstall. @MichaIng has done some more indepth improvements to the installation code: https://github.com/Fourdee/DietPi/issues/2187#issuecomment-435536545

I'll mark this as closed now, however, we can improve and add more software to this in the future as needed.

MichaIng commented 5 years ago

@Fourdee I will reopen. Want to go through all software titles and complete the list above by times, sort of ongoing project 😉. If closed, I might forget 😅.

Fourdee commented 5 years ago

@MichaIng

Yep, no worries 👍

Configs currently backed up by G_BACKUP_FP, before we modify/replace:

    Line 6347:          G_BACKUP_FP /etc/X11/xorg.conf
    Line 7235:          G_BACKUP_FP /etc/apache2/apache2.conf
    Line 7299:          G_BACKUP_FP /etc/nginx/nginx.conf
    Line 7332:          G_BACKUP_FP /etc/lighttpd/lighttpd.conf
    Line 8374:          G_BACKUP_FP /etc/mpd.conf
    Line 8437:          G_BACKUP_FP /etc/proftpd/proftpd.conf
    Line 8450:          G_BACKUP_FP /etc/samba/smb.conf
    Line 8465:          G_BACKUP_FP /etc/vsftpd.conf
    Line 8623:          G_BACKUP_FP $HOME/.config/cava/config
    Line 8665:          G_BACKUP_FP ~/.config/mopidy/mopidy.conf
    Line 8771:          G_BACKUP_FP /etc/minidlna.conf
    Line 9047:          G_BACKUP_FP $G_FP_DIETPI_USERDATA/deluge/.config/deluge/core.conf $G_FP_DIETPI_USERDATA/deluge/.config/deluge/web.conf
    Line 9659:          G_BACKUP_FP /etc/fail2ban/fail2ban.conf /etc/fail2ban/jail.conf
    Line 9952:          G_BACKUP_FP /etc/dhcp/dhcpd.conf
    Line 9998:          G_BACKUP_FP /etc/hostapd/hostapd.conf
    Line 10097:             G_BACKUP_FP /usr/local/etc/shairport-sync.conf
    Line 10470:             G_BACKUP_FP /etc/mineos.conf
    Line 10547:             G_BACKUP_FP /home/qbittorrent/.config/qBittorrent/qBittorrent.conf
    Line 10904:             G_BACKUP_FP /var/lib/dietpi/dietpi-software/installed/aria2.conf
    Line 11343:             G_BACKUP_FP /etc/motioneye/motioneye.conf
    Line 11690:             G_BACKUP_FP $G_FP_DIETPI_USERDATA/nzbget/nzbget.conf

MichaIng commented 5 years ago

@Fourdee Jep, will make use of this too, makes sense to have consistent method + output for this 😃.

Hmm, I think I will move all of this together with current software versions (https://github.com/Fourdee/DietPi/issues/1340) into a Wiki page. So we have a general overview outside of issues, where we and users can quickly check the versions/sources we install and how reinstalls are handled.

Fourdee commented 5 years ago

@MichaIng

Jep, will make use of this too, makes sense to have consistent method + output for this 😃.

Good stuff 👍

I will also add a /var/tmp/dietpi/logs/G_BACKUP_FP_list file, so users can see which files have been backed up etc.

Fourdee commented 5 years ago

@MichaIng

Hmm, I think I will move all of this together with current software versions (#1340) into a Wiki page. So we have a general overview outside of issues, where we and users can quickly check the versions/sources we install and how reinstalls are handled.

Yep sounds good 👍

Looking back now, wish I used WiKi instead of a forum lol. Too much work to move everything now though.

MichaIng commented 5 years ago

@Fourdee

Looking back now, wish I used WiKi instead of a forum lol. Too much work to move everything now though.

There are many online BBCode => Markdown translators, that should do the job. It is as well good to have info in own forum, e.g. when Microsoft influence forces us to move to another Git provider.

Markdown => BBCode translators do not work that well, but for consistency might be beneficial as well to port Wiki pages and/or other info from issues back to our forum.

The info from the table I mentioned above could be added to the forums software page. But yeah much work 🙈.

Fourdee commented 5 years ago

Ok, G_BACKUP_FP now implemented.

We just need to remember to roll this out as required in future.

Will mark as completed.

MichaIng commented 5 years ago

@Fourdee Jep, better/individual config handling is an ongoing process implemented, when we touch related install code anyway.

MichaIng / DietPi