Cannot re-install an OS due to partition not big enough though I should have free space

[garret]

I have PINN installed on a RaspberryPi 3B+ with two OSes: DietPi (default) and Batocera. The internal SD Card with PINN installed is 32GB. The PINN build is v3.3.2.

I have an old version of Batocera and would like to install the latest one. Usually, I have been just re-installing that OS from the PINN boot menu but cannot manage anymore as I get an error message.

In particular, this is the screen that shows my attempt to re-install Batocera. It should be noted how the Destination drive field is empty and there is no option available to select. Don't remember if that is normal to be empty.

This is instead the error I get:

Do you know how to fix this issue?

PS: I really love PINN, sorry I had to say it.

[lurch]

Have you tried using the latest version of PINN?

[garret]

@lurch do you think that will solve the issue or is just a proposal? I am asking because I would have preferred to leave PINN installation intact as it has been working fine for what I need. I am worried to break even more things by updating ...and don't really know how to do it either

[procount]

I don't know much about Batocera, because it is not one of my conversions. But looking at the partition sizes, it doesn't seem to use the standard layout, requiring a 3GB 1st partition and then an additional spare partition, instead of the usual small Boot partition and a larger root partition. So I suspect the 1st partition is just not big enough to contain the latest version, as the error message says. Typically we only add 100MB to the boot partition to allow for some extra space, but in the case of Batocera, this should perhaps be a bit more for future expansion.

It is usual for the destination drive box to be empty if you don't have any USB drives attached.

I can think of a couple of options to solve this:

1. Backup DietPi to a USB stick. Then Install your backup of Dietpi alongside the new Batocera version. This will force PINN to repartition the SD card to the correct new sizes. Make sure you have a good backup of DIetpi before doing this. Or better still, install your backup and batocera to a new SD card. You can take this opportunity to upgrade PINN to the latest version too.
2. Use Gparted (in Dietpi if it supports it, or on another Linux system) to resize your Batocera partition is that it is big enough to support the newer version.
3. If Batocera is the second OS, it may be easier just to delete the Batocera partitions and recreate them with a larger 1st partition. (You can do it if it's the first OS aswell, but you have to be more careful not to overlap the 2nd OS.

[garret]

@procount Thank you very much for your message. Option 3 could result the "easiest" for me as I only care about the DietPi installation, so it would be fine to remove completely the Batocera OS and reinstall it. As I remember, I also installed Batocera as a second OS. I have some questions then:

1. How can I check I installed Batocera as second OS?

2. How can I backup DietPi? Or actually I would prefer to just backup all my 32GB SD card at this point. Is there an easy way? I checked on the Wiki but could not find anything about. I was thinking to use Win32DiskImager to "read" the entire SD card and save it as 32GB .img file. However, I can see that when plugging the SD card in my Win10 installation, Win32DiskImager sees two drives (D: and E:) and don't really know which one to choose.

3. How can I remove the Batocera OS? In the top menu bar from my first picture I cannot see any "Remove" button.

Thank you very much for the support!

[procount]

1. Press ctrl-alt-f2 to get to a new terminal and login with root/raspberry. Then type cat /settings/installed_os.json. The resultant structure will have 2 sections - one for DietPi and another for Batocera. Note the ordering of the partitions for each OS, or just paste the contents here. (Press ctrl-alt-f1 to get back to PINN).
2. See https://github.com/procount/pinn/blob/master/README_PINN.md#backup (This is the main instruction manual that you should read to get the best out of PINN). You will need a USB drive to store the backup on. Win32DiskImager is not preferred because you may not be able to restore the image to another SD card if it is slightly smaller. Alternatively, you can use PINN's Clone SD card to copy the whole SD card to another SD card in a USB reader. But the backup OS option is probably the best.
3. You cannot remove OSes individually. You can only reinstall them or replace with another one. If you took a backup of DietPi and then INSTALL your backup and the new Batocera, it will firstly WIPE anything that was previously there. For safety, you could get another SD card, install the latest PINN version on it, then install your Dietpi Backup and Batocera to that, thus preserving your old installation.

[lurch]

It is usual for the destination drive box to be empty if you don't have any USB drives attached.

Sorry, I didn't realise that! That was the only reason I was suggesting an upgrade :confused:

Also, one of the messages in your screenshots says "All existing data on the SD card will be deleted", so I didn't realise that you were actually planning to keep your DietPi installation. @procount Perhaps that dialogue should say something like "All existing data on partitions X and Y of the SD card will be deleted" ?

[procount]

@lurch - a classic case of over re-use of a dialog box! Will try to remember to fix it on next iteration.

[garret]

It is usual for the destination drive box to be empty if you don't have any USB drives attached.

I didn't really pay much attention but I wanted to state that I actually have two usb drives attached to my raspberry pi (both formatted as ext4). So isn't it strange that PINN shows an empty field?

In any case, then at this point I think I will try the backup option just for DietPi, reinstall a new version of PINN, restore DietPi and then install Batocera. I will let you know as soon as I manage to do that :)

[procount]

I didn't really pay much attention but I wanted to state that I actually have two usb drives attached to my raspberry pi (both formatted as ext4). So isn't it strange that PINN shows an empty field?

Do either of them have an /os folder on them?

[garret]

@procount no, and I guess that should explain why that field is empty then.

I will try to do the above backup and restore operation before the end of next week and report if I encounter any problems. Thank you in the meanwhile.

[garret]

Finally, I had time to do the backup. First I created a backup with win32diskimager of the whole sd card and then I created a folder called "os" in a usb stick (64GB) formatted as FAT on Windows. PINN recognized this usb stick and I backed up the DietPi os.

I formatted the SD card and copied the files of the latest PINN version. I booted PINN and selected the DietPi backup.

However, it seems like the installation process gets stuck. I can see the speed going down (and thus the time going up). So no more than 169 MB are copied.

Nothing happens after coming back after 2 hours (just time of installation goes insanely high).

I tried to re-do the process but always end up in the same issue. Unfortunately, I didn't have another SD card to install PINN and test my DietPi backup.

Do you know why it is behaving like this?

I can access from Windows to the os folder and see the folder containing the DietPi backup. For the moment I managed to restore the previous backup by restoring the image created with win32diskimager on the SD card. So, theoretically now it is like I'm again at the starting point with my DietPi and Batocera systems working but would like to update batocera by reinstalling it.

[procount]

Do you know why it is behaving like this?

No. It's odd that it should stop installing like that. Could you try formatting your USB stick as NTFS, add the /os folder, repeat the backup, and then see if that can be installed?

[garret]

@procount I did as you suggested but still it doesn't work. In particular:

1. I updated PINN by using the automatic updater on my 32GB sd card (so now have the latest version);
2. Formatted a 32GB usb stick as NTFS and created a "os" folder inside with Windows;
3. Backed up DietPi from my 32GB SD card to this 32GB usb stick by using the "Backup" function of PINN;
4. Borrowed a 16GB SD card (so not the original one of 32GB), formatted it as FAT and copied PINN files by using Windows;
5. Tried to install the backup through PINN interface but now it gets stuck at this screen:

Actually, the system is frozen. I cannot even move the mouse and this is valid also after an hour.

[lurch]

How bizarre that one should get stuck at 169MB and the other gets stuck at 162MB ! Have you ever had any other stability / power-supply issues with your Pi?

Although it seems a bit weird that your DietPi backup-size has increased from 4452MB to 5970MB in just a few days? :confused:

[garret]

How bizarre that one should get stuck at 169MB and the other gets stuck at 162MB ! Have you ever had any other stability / power-supply issues with your Pi?

@lurch I actually now retried for a third time and the installation went to 100%. However, an error message appeared at the end:

I don't know how to see the debug though 😕

Although it seems a bit weird that your DietPi backup-size has increased from 4452MB to 5970MB in just a few days? 😕 I think that is normal as in these few days did many things on my DietPi installation, so I am not that surprised of the backup size increased.

I never had any stability or power issue with my raspberry pi.

[procount]

I don't know how to see the debug though

https://github.com/procount/pinn/wiki/Troubleshooting - especially part 5/6 In summary, press ctrl-alt-F2, login with root/raspberry and type less /tmp/debug and scroll through the debug log til the end.

I may have some more time soon, so I can try to replicate this issue with DietPi

[garret]

@procount today I found some time to copy both the dmesg and debug logs as reported in the Troubleshooting page. I attached them here:

• debug.txt
• dmesg.txt

I really hope these files can help to understand the issue. Please let me know if you need more (and thank you very much again in the meanwhile!).

[procount]

Hmm. This seems to be the problem:

"./mnt/dietpi_userdata/docker-data/overlay2/648960dc58ea184458373e4ed2f8c1be36b66bfb5195d87f4d0a94f34e8e5da5/diff/bin/sh.real: Hard-link target './mnt/dietpi_userdata/docker-data/overlay2/648960dc58ea184458373e4ed2f8c1be36b66bfb5195d87f4d0a94f34e8e5da5/diff/bin/sh' does not exist.

So a problem with hardlinks and how they treated in bsdtar. Needs some investigation.

[garret]

So a problem with hardlinks and how they treated in bsdtar. Needs some investigation.

@procount On DietPi I am always running three containers on Docker. Maybe it could help to stop them, power off the system/DietPi and do a new backup of the DietPi with PINN?

[procount]

You can try. If no good you could try removing the docker containers before backing up and restore them separately later.

[lurch]

Yeah, probably some weird interaction between bsdtar, hardlinks, and the real/virtual filesystem links (overlayfs?) used by the files in the Docker containers. https://www.google.com/search?q=bsdtar+hard-link+target+does+not+exist has a few similar-sounding problems, but no definitive reference.

[procount]

I understood bsdtar would handle hardlinks, but to do so it would need to ensure the target file is stored in the tar file before any hardlinks reference it. This must require reordering of the files as they are archived and the use of a lookup table.

But it is failing because the target doesn't exist. Could this be because the tar file is not ordered correctly? Or because the target is on a different filesystem? Maybe because of an overlayfs as @lurch suggests?

I looked at some of the Google refs and they mostly talk about printing an error message, rather than fixing the problem

[garret]

I can see I have many folders on /mnt/dietpi_userdata/docker-data/overlay2/:

Maybe I can cancel those two files? How do I know if they are being used on my 3 running containers? Because maybe I don't need them (or at least I hope).

Or at least it would be nice which container created such files just to know.

[procount]

I'm afraid I have no idea about docker files. Maybe @lurch knows more?

You could try checking the files exist in the tar file. Something like:

bsdtar -tvf root.tar.xz | grep diff/bin/sh

(You may have to adjust that as required as I'm not at my PC or pi atm.)

[lurch]

I understood bsdtar would handle hardlinks, but to do so it would need to ensure the target file is stored in the tar file before any hardlinks reference it. This must require reordering of the files as they are archived and the use of a lookup table.

AIUI that would be the case for symlinks, but not for hardlinks. With hardlinks there's no "target file" because all filenames point to the same inode, so I assume the first file that bsdtar finds with num_links > 1 becomes the "target file", and then any other files with (num_links > 1 and) the same inode then become "hardlinks to" that target file inside the bsdtar archive? :shrug:

I'm afraid I have no idea about docker files. Maybe @lurch knows more?

I've got vague docker knowledge, but not very much. However it looks like docker-data/overlay2/648960dc58ea184458373e4ed2f8c1be36b66bfb5195d87f4d0a94f34e8e5da5/diff/bin/sh probably maps to /bin/sh (i.e. the default shell interpreter) inside the OS being run by docker, so deleting that file would probably cause that docker-container to fail to boot! Inside each of your docker containers, what do each of:

ls -l /bin/sh
ls -l /bin/sh.real
stat /bin/sh
stat /bin/sh.real

say?

https://www.google.com/search?q=docker+overlayfs+hardlinks appears to have lots of info, but I'm afraid I don't have time to wade through it all at the moment.

[garret]

@lurch Finally I had a bit of time to run the commands you mentioned. I have three containers but two of them reported the following error when trying to ssh into them:

OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: \"/bin/bash\": stat /bin/bash: no such file or directory": unknown

Instead I successfully sshed into the last one and this is the result of your commands:

root@30d23bf098c2:/# ls -l /bin/sh
lrwxrwxrwx 1 1000 users 4 Jan 24  2017 /bin/sh -> dash
root@30d23bf098c2:/# ls -l /bin/sh.real
lrwxrwxrwx 1 1000 users 4 Jan 24  2017 /bin/sh.real -> dash
root@30d23bf098c2:/# stat /bin/sh
  File: /bin/sh -> dash
  Size: 4               Blocks: 0          IO Block: 4096   symbolic link
Device: 29h/41d Inode: 404420      Links: 1
Access: (0777/lrwxrwxrwx)  Uid: ( 1000/ UNKNOWN)   Gid: ( 1000/   users)
Access: 2017-01-24 06:16:56.000000000 +0100
Modify: 2017-01-24 06:16:56.000000000 +0100
Change: 2019-10-11 11:56:42.664856141 +0200
 Birth: -
root@30d23bf098c2:/# stat /bin/sh.real
  File: /bin/sh.real -> dash
  Size: 4               Blocks: 0          IO Block: 4096   symbolic link
Device: 29h/41d Inode: 270936      Links: 1
Access: (0777/lrwxrwxrwx)  Uid: ( 1000/ UNKNOWN)   Gid: ( 1000/   users)
Access: 2017-01-24 06:16:56.000000000 +0100
Modify: 2017-01-24 06:16:56.000000000 +0100
Change: 2019-10-11 11:56:42.964856140 +0200
 Birth: -
root@30d23bf098c2:/# 

Let me know if I can be of more help...

[lurch]

Hmmm, so (in that particular container) /bin/sh and /bin/sh.real are both separate symlinks to the /bin/dash binary. I'm afraid I've got no idea why bsdtar would be complaining about hardlinks then :confused:

I still suspect the problem is likely to be some weird interaction between bsdtar and docker and overlayfs, but I've no idea what else to suggest. :man_shrugging: :penguin:

[procount]

I suppose when PINN is backing up the OS, the partitions are not mounted, and it is only backing up one filesystem. So I guess at this point the overlayfs is not mounted, so some links may not be valid :man_shrugging:

[garret]

Hmm so any suggestion on what could I do? You think that stopping the containers might not be enough when I backup? I might encounter still the same issue?

[procount]

My comment was only speculation as I have no experience using Docker on DietPi. But my understanding of Docker is that it is supposed to be portable between machines. Can you not do a fresh install of DietPi and then pull down your docker configurations from their websites and otherwise rebuild them?

[mrdc]

I have similar issue and it looks like it’s because of errors in partition table. Batocera requires big boot partition, which differs from the raspbian and other OSes where we have small fat32 boot partition and large ext4. Checking boot partition for the OS you want to replace for errors/formatting it/resizing may help to fix the issue.