Local configuration changes silently lost

[Paraphraser]

In #108 I noted that a custom modification I had made to:

~/IOTstack/services/mosquitto/mosquitto.conf

to append this line:

log_timestamp_format %Y-%m-%dT%H:%M:%S

had been lost. I am not sure when this happened. My guess is four or five days ago when I did a "git pull origin master" followed by a menu run with full overwrite. I had not been into the mosquitto log to chase a problem since then so I hadn't noticed the format reversion.

All up, no harm, no foul.

But it does raise an issue for me because local changes have been silently lost. To me, that feels like a Bad Thing.

Any thoughts on how to deal with this kind of problem? Perhaps something based on:

cmp --silent template existing || echo "files are different"

where the RHS copies "existing" to "existing.bak", bungs out a warning that "existing" is being overwritten, points the user to "existing.bak", and suggests manual resolution of any conflicts.

I reckon a warning is always better than silence. What say you?

[gcgarner]

Hi Phil

This is an extract from my menu script. When rebuilding the stack you are given 3 options

• Do not overwrite
• Preserve Env and config files
• Pull full template

    if [ -d ./services/$1 ]; then
        #directory already exists prompt user to overwrite
        sevice_overwrite=$(whiptail --radiolist --title "Overwrite Option" --notags \
            "$1 service directory has been detected, use [SPACEBAR] to select you overwrite option" 20 78 12 \
            "none" "Do not overwrite" "ON" \
            "env" "Preserve Environment and Config files" "OFF" \
            "full" "Pull full service from template" "OFF" \
            3>&1 1>&2 2>&3)

        case $sevice_overwrite in

        "full")
            echo "...pulled full $1 from template"
            rsync -a -q .templates/$1/ services/$1/ --exclude 'build.sh'
            ;;
        "env")
            echo "...pulled $1 excluding env file"
            rsync -a -q .templates/$1/ services/$1/ --exclude 'build.sh' --exclude '$1.env' --exclude '*.conf'
            ;;
        "none")
            echo "...$1 service not overwritten"
            ;;

        esac

If one was to select the preserve env files your changes you made to your conf and env file via the --exclude '$1.env' --exclude '*.conf'. However I explicitly asked you to pull a full template when we were trying to sort out my botchup of the mosquitto container volume mapping.

I think you do have a good point and that I should add a confirmation in the Full template path

As a sidenote you will see that in the .gitignore file that i explicitly exclude the services and volumes directories so that my and any other contributors services don't get bundled into the project by accident

[Paraphraser]

I am still trying to wrap my brain around these three options. Perhaps I can ask the question like this. What is the canonical use case for each option?

To my way of thinking, “do not overwrite” sounds like a no-op. It is the option I choose for containers A, B and C if I’m running the menu with the intention of only doing something to container D. “Do not overwrite” is the option you have when you want to do nothing. Yes?

“Pull full template” sounds like it should mean “rebuild from scratch”, only we’ve just worked out that it doesn’t actually do that. I still had to manually clobber the mosquitto container then run “up” so it could be recreated. Would not the same effect have been achieved by “do not overwrite” (or, indeed, not even running the menu at all) followed by a remove and an up?

So, if pull full template isn’t a nuke-and-rebuild, when is it indicated (aside from when you tell me to do it and I blindly obey)?

And that brings us to the “preserve” option. When is it likely that to be needed, rather than the other two?

I suppose what I’m trying to get you to do is to see it from the user’s perspective rather than the designer’s perspective. Does that make sense?

[gcgarner]

I think it is important to remember that the Build Stack option only builds the docker-compose.yml file

There is a distinction between the yaml file and the volumes.

What i attempted to do with the 3 options is give people more flexibility. Because the compose file is divided up into the contents of the service.yml file and the env and conf files.

If you are happy with the file then leave it as is.

if you have altered the env or config then choose the preserve. this catches when people change passwords, ports and other configs in the env files. one case is if you want to run grafana behind a reverse proxy you have to edit the env file or it wont work. If you didn't preserve the config then every time you rebuild the stack you would lose all those changes

finally the full template. This is if you need to get back to the as build template without any changes. Maybe you botched the config file or broke something in the service definitin (or someone didn't do the config correctly in the first place). It will give you the fresh start however will keep your data.

unfortunately as we saw with some of the containers, they can sometimes ignore the env file and would require the nuclear option of deleting the volume and starting from a clean slate

[Paraphraser]

I grok the "full template" option but the remainder of your reply makes me think that the safest course is always to choose preserve - otherwise it sounds like I have to remember which ENV/config files I have changed and which I haven't.