Executing the backup script gives errors/gets stuck (wrong ownership)

[IhsenBouallegue]

When I try to execute the backup script, I get the following message

No matter what I type, it won't proceed. I don't think this is normal or expected behaviour.

[Paraphraser]

Please read what is sudo.

And please don't think "I already know what sudo is - I can skip that". It's a page in the IOTstack Wiki that you really need to read before you go on.

What you are trying to do is run the backup script as the root user. It doesn't expect that, which is why there's a problem.

Many (but not all) IOTstack scripts have a check at the start to detect this and they will exit with the message "this script should NOT be run using sudo". It would be better if all the scripts had this treatment but it's a work in progress.

If you read that link above carefully, you should come away with a better appreciation of when you should not (most of the time) and should (rarely) use sudo to run a script.

What bothers me most about your screen shot is the root@raspberrypi prompt. That suggests you've either enabled the root account and then logged-in as root, or have done something like a sudo -s to get a privileged shell.

And then I wonder whether you're in the habit of doing that a lot when working with IOTstack.

IOTstack is intended to be installed in the home directory of a non-privileged user (eg the default user "pi"). Its scripts are intended to be run without sudo. The docker and docker-compose commands are intended to be run without needing to use sudo.

The default user (eg "pi") is a member of the so-called "sudoers" group which lets the unprivileged user (pi) escalate to root privileges, command by command. Most scripts take advantage of that by using sudo to execute certain commands when it is necessary to do so. That's why the link I gave you above suggests using grep to search scripts for "sudo" because that tells you the script should not be invoked using sudo:

$ grep -c sudo ~/IOTstack/scripts/backup.sh 
3

Similarly, on a proper installation of IOTstack, the default user ("pi") will be a member of the docker group. That provides all the permissions needed to run both docker and docker-compose. If you try running those commands and you get errors suggesting the need for sudo, it's actually telling you that your installation is somehow mucked up.

A lot of things flow from improper use of sudo. If you clone IOTstack from GitHub using sudo, quite a few things that should be owned by "pi" will be owned by "root" and then nothing will work properly. You use sudo to overcome those problems and that just compounds the bigger problem - a whole cascade of issues.

Bottom line: if you are finding that you need sudo a lot and/or you spend a lot of time inside a privileged shell, something may be wrong with your installation. You might be better advised to start over. If you decide to do that, reply and I'll give you some more tips.

[IhsenBouallegue]

Hey @Paraphraser Thanks for the detailed explanation. I am no linux expert, so please bear my ignorance. However, I do not use sudo with IOTstack. In fact, I barely used any command outside the menu.sh. This time I resorted to running the command manually because running the backup from menu.sh prints permission denied errors and is a bit bugged. Running the command normally without sudo prints the same, but without backup completed at the end. And that is why I used sudo because it explicitly said Permission denied

[Paraphraser]

Hi,

Let's rewind a bit. I'll try to explain what's going on.

In your very first screen shot, the system prompt is:

root@raspberrypi:/home/pi/IOTstack#

That says you are running as root. The most likely way to get into that state is to login as "pi" and then run these commands:

$ cd IOTstack
$ sudo -s

You can also replace sudo -s with sudo bash. It's exactly the same. Both commands mean, "become root and start a shell".

The only difference between the last line of my screen shot and the first line of yours is the system name (your Pi is named "raspberrypi" while mine is named "sec-dev").

Also look at the last character of each prompt. In the first two lines you'll see "$" which means "running as a non-privileged user". In the third line you'll see "#" which means "running as root". So, both the root@ and the # confirm you are running as root in your first screen shot.

Now, let's try an experiment. In the following the $ at the start of the first three lines is intended to convey "run this command as an unprivileged user", while the # in the remaining lines means, "this command is being run by a privileged user". This structure where the expected prompt is shown is a common convention and is intended as a sanity check to avoid running commands with the wrong level of privilege.

Anyway, here we go:

$ cd
$ touch user-file.txt
$ sudo -s
# touch root-file.txt
# mkdir root-dir
# touch root-dir/root-file.txt
# exit

In words:

1. cd with no arguments changes to your home directory.
2. touch creates an empty file, owned by the current user (pi) and with default permissions (which we will see in a bit).
3. sudo -s starts a shell with root privileges. The prompt changes to #.
4. Because we are running as root, the file created by touch will be owned by root.
5. Create a directory. We are still running as root so this directory will also be owned by root.
6. Create a file inside that directory. We are still running as root so that file, too, will be owned by root.
7. The exit command (a synonym for control+d) drops out of the root shell and we're back to being "pi" and you'll see the prompt has gone back to $.

Now let's assess the situation:

$ ls -ld . root-dir
$ ls -l user-file.txt root-file.txt root-dir/root-file.txt

The d on the -ld means "show the information about the directory rather than its contents". In words:

1. The . directory (the current working directory which is also the user's home directory) is owned by "pi".
2. The root-dir directory is owned by "root".
3. The two root-file.txt files are owned by "root".
4. The user-file.txt is owned by "pi".

Now, while running as "pi", let's try deleting all of that:

$ rm user-file.txt
$ rm root-file.txt
$ rm root-dir/root-file.txt
$ rm -rf root-dir

Step by step:

• "pi" owns user-file.txt so we expect to be able to delete that file. The Unix convention is that silence means success, so no error means the file has been deleted.
• "root" owns root-file.txt so, all other things being equal, "pi" should not be able to delete that file. Unix will ask whether we want to override. If we say "yes" then the file is removed. How come? It's because the file is in a directory owned by "pi". The owner of a directory gets special dispensation.
• "root" owns both root-dir and the root-file.txt inside that directory. We get asked if we want to override but the operation fails anyway because we don't own root-dir. No special dispensation.

• "pi" can't remove root-dir recursively for the same reason (the root-file.txt inside it can't be removed). The only way to clean up is to use sudo:

  $ sudo rm -rf root-dir

The point of all this is not to bore you rigid with Unix intricacies but to try to set the scene to help you understand what's going on in your second screen shot.

Let's just consider the very first line where the error is calling out line 86 in backup.sh. We can find line 86:

$ head -86 ~/IOTstack/scripts/backup.sh | tail -1
echo "" >> $LOGFILE

In words, the echo command is writing an empty string to the file path held in the LOGFILE variable. You can see that file path in the next part of the error message:

./backups/logs/backup_2023-01-23_1456.log

The backup.sh script expects the working directory to be ~/IOTstack so that . at the start expands to:

/home/pi/IOTstack/backups/logs/backup_2023-01-23_1456.log

I'm going to attempt to simulate all of that, like this:

$ cd ~/IOTstack
$ LOGFILE=./backups/logs/backup_2023-01-23_1456.log
$ echo "" >> $LOGFILE

The reason it failed is because I don't actually use the backup.sh script so, although the backups directory exists on my system, the logs sub-directory does not. I can fix that by creating logs by hand and trying again:

Now the echo command works, which means line 86 in the script would work too.

But now I'm going to stuff everything up by:

• removing the logs sub-directory and its contents;
• becoming root and creating logs again, this time owned by root; and
• trying exactly the same command again.

Exactly the same error as you got, right? The only difference between success and failure in the last two screen shots was the use of sudo to create the logs sub-directory.

So, that's the basic problem. At some point you were running as root and "did something" that created the logs sub-directory with root ownership, and now anything non-root that tries to create files inside that directory is blocked.

Your second screen shot also includes problems with the path:

./.tmp/backup/ …

and that's probably for the same reason.

My guess is that your entire IOTstack sub-directory structure is probably peppered with consequential problems of this kind but, without having access to your system, I can't be sure how widespread the problem actually is.

Assuming you don't want to nuke your entire system and start from scratch, I'll give you some steps that should fix most (hopefully all) of the damage.

Take down your stack:

$ cd ~/IOTstack
$ docker-compose down

Move to your home directory then rename IOTstack to get it out of the way:

$ cd
$ mv IOTstack IOTstack.old

Get yourself a clean clone of IOTstack from GitHub:

$ git clone https://github.com/SensorsIot/IOTstack.git IOTstack

Move into the old IOTstack directory:

$ cd ~/IOTstack.old

Now we're going to use sudo to change ownership on a few key files and folders. We are just forcing ownership to what it should be (what it would be if sudo had never gotten in on the act in the first place):

$ sudo chown -R "$USER:$USER" docker-compose.yml services
$ FILES=".env compose-override.yml docker-compose.override.yml"
$ for F in $FILES ; do [ -f "$F" ] && sudo chown -R "$USER:$USER" "$F" ; done

Now we'll copy the items we just fixed into the clean clone of IOTstack we just downloaded from GitHub:

$ cp -a docker-compose.yml services ~/IOTstack/
$ for F in $FILES ; do [ -f "$F" ] && cp "$F" ~/IOTstack/ ; done

Also make a note of the size of your persistent store:

$ sudo du -sh volumes

Now you need to make a decision about your persistent store. I'd recommend making a copy but I don't know how much free space you have on your SD/SSD so the alternative is to make a move.

The reason I recommend a copy is because it gives you a fail-safe. If something mucks up, you can remove the clean clone of IOTstack and start over. I've done what I'm describing here quite a few times so it's not particularly unsafe. I'm just being ultra-cautious. But, if you don't have sufficient space on your SD/SSD then a move is your only option.

The result of the du -sh command above gave you the size of the persistent store. You can get the amount of free space on your SD/SSD by running:

$ df -h /

Hopefully you're still in the old IOTstack directory but, just to be on the safe side, make sure:

$ cd ~/IOTstack.old

If you decided to:

• EITHER copy your persistent store:

  $ sudo cp -a volumes ~/IOTstack/

  Depending on size of volumes and the speed of your SD/SSD, this can take a fair amount of time so be patient. If you're wondering, the combination of sudo and the -a flag means the copy command (cp) preserves existing ownership and permissions.

• OR move your persistent store:

  $ sudo mv volumes ~/IOTstack/

  This will be almost instantaneous. It also preserves existing ownership and permissions.

Then you should be able to bring up your stack again:

$ cd ~/IOTstack
$ docker-compose up -d

If docker-compose doesn't work and you get the urge to use sudo, please tell me. It means docker and docker-compose aren't installed correctly. I can help you fix that. Don't use sudo to force it.

What has all that accomplished? As the IOTstack menu and other scripts run, they produce all manner of side-effects as they create directories and files. A clean clone of IOTstack has reset everything to "factory fresh". We've ensured docker-compose.yml and services (and other key files like .env if they exist) have the correct ownership. In general, the volumes folder looks after itself. Overall, the entire sequence of commands is the closest thing we have to a "master reset" switch for IOTstack.

Once you're happy that the new stack is working properly, you will probably want to run the menu at least once so it can do its setup housekeeping:

$ cd ~/IOTstack
$ ./menu.sh

If you get a screen about "large update", ignore everything it says and answer "no".

There will be a small delay while the Python virtual environment is re-created. After that, everything should be back to normal.

---

Eventually, you'll get to a point where you have sufficient confidence in the "clean clone" of IOTstack that you'll realise you no longer need the old directory. You can clean it up with:

$ sudo rm -rf ~/IOTstack.old

That's a very powerful and dangerous command if you get it wrong so please make sure you get it exactly right before you press return.

---

Earlier, I said I don't use the backup.sh script. What I use instead is:

• IOTstackBackup

The primary advantage of IOTstackBackup is that you don't need to stop your stack just to run a backup. The only "gotchas" are if you run either PostgreSQL or Subversion containers - those are just ignored by IOTstackBackup.

Also, the scripts in IOTstackBackup will not actually let you make the mistake of running them as the root user. That doesn't mean they won't chuck up errors if some other inappropriate use of sudo has somehow mucked up the expected structures inside the IOTstack directory but they won't be the cause of it.

---

Please don't feel too bad about all this. You are far from the first person to run into strife with sudo (me included). That's why that page in the Wiki was written in the first place. The need to use sudo from time to time is a well-known Unix weakness.

In macOS, Apple has tried to hide a lot of the details by having the system ask for permission to escalate privileges when necessary, as opposed to expecting every user to "just know" when they have to use sudo but the underlying problem is still there.

I do recommend that you adopt the approach proposed in that Wiki page. Try commands without sudo first, and only use sudo to force the issue if the first attempt fails. It doesn't guarantee you'll never hit this kind of problem again, just reduces the chances a bit.

Anyway, I hope this all helps. You might even know a bit more about Unix than you ever wanted so. Sorry about that.

---

If you're wondering, the main reason I write long-winded and detailed answers to issues is because everything saved on GitHub persists. The next time this kind of problem comes up, I can just point the person to this issue.

[IhsenBouallegue]

@Paraphraser I am out of words. Thank you so much for every detail. Sorry if I took too long to reply, it took quite a while and a few reads to understand and execute. I followed your instruction to the letter. Sadly, the problem persisted. But because I have a better understanding and not only a list of commands, I debugged a bit. I checked the ownership of my backup directory (in the new IOTstack), turns out, it is root for some reason (probably a fault of mine). I changed it like we changed the other files and surprise surprise that solved it. But that opened my eyes to how messed up my setup might be. I was going to do a fresh install anyway because I bought an SSD for IOTstack (hence the backup). I am done living in the fear of an SD card fail 😆 But I am going to do it right this time and use the PiBuilder tool (which I discovered through your github!) That way, I make sure everything is set up the right way.

Again, thank you for the detailed explanation and the awesome tools you wrote ❤️. This made me love IOTstack more. And especially trust myself around it more 😛

I am closing this issue as the original problem was solved 👍

[Paraphraser]

Actually, that one is pretty rare and possibly the result of a sudo git clone. But, I did actually think of that, which is why I said to move the old IOTstack out of the way and clone a fresh copy without sudo. If that fresh copy somehow got root ownership then, I agree, something is so hosed-up somewhere that it probably really is time for the rebuild you're planning.

By the way, you should be able to install IOTstackBackup on your existing system and use that to take a "before" snapshot. A PiBuilder run includes IOTstackBackup (and clones IOTstack) so, basically, you get to the end of the PiBuilder run, execute the iotstack_restore command, and then tell your stack to come up.

[IhsenBouallegue]

@Paraphraser that's exactly what I did, and now it works perfectly and as expected 👍 Thank you again 🙏