Repartition (SD) disk and shrink / to minimum

[mstormi]

Current Raspbian is making full use of the available space of SD cards by extending the / filesystem to match its size. This comes at a cost: Whenever a user needs to restore the disk contents (using dd to copy it or Amanda amrestore or whatever), his new SD card needs to be at least as large as the previous (source) one. Given the large number of people with corruption problems running Pis on SD and given the number of different cards on the market, you can run into this problem fast and at some point, your only choice is a full openHABian reinstallation (image re-deployment). I've shrunk my / following http://www.netzmafia.de/skripten/hardware/RasPi/SD_Karte_verkleinern.html and am happy I did. I have been in need of restore two times since and even multiple more times before.

In order to avoid running into this, I suggest to minimize the root partition and mount the remaining disk space as separate partitions. Important part is to have the expendable data in the last partition i.e. at the end of the medium thus whenever you need to restore to a smaller medium, this will still succeed and all you're potentially loosing is expendable data.

I would like combine this with the suggested filesystem reorg from issue #128:

• shrink / to a defined minimum size so it fits even on smallest cards (4GB?)
• create 2 partitions out of the excess medium space and mount them as /volatile/essential and /volatile/expendable and move all logs (system, OH, 3rd party tools as well) and OH persistence directories there. We can discuss the details which dirs to move and how (loopback mount or move+link) and maybe adapt this over time. See #128 for a proposal.

But this sort of restructuring is a one-time prerequisite. Since it is a somewhat breaking change I think it would be a good idea to introduce as part of a new image version only so we avoid running into trouble with backward compatibility. Note another advantage of this restructuring: it will also prepare for an easy future move of these directories over to a NAS or USB mount or even tmpfs. That'll remain a user decision if and where to move, but this filesystem rework is a prerequisite so it should become the openHABian default.

Totally unsure for now though how to do it best. Is it easier to create a modified Raspian image in the first place to contain that already? Inject those changes into Raspbian installation routine ? Or should we ask for in openhabian-setup and perform all the moves there ? That way we could benefit from this on platforms other than Pi, too , but it's possibly quite tricky and prone to error. Wdyt ?

[ThomDietrich]

Regarding the first part: If you've been using a 16GB card so far, you can simply replace it by a 16GB or bigger card. Chances are you can buy a bigger card as you are using right now for ~10€. I don't see an issue here.

A small partition will most certainly introduce new issues. Exceeded capacity or increased write cycles are the obvious ones...

You gave a lot of details but I still didn't understand the benefit of over-complicating things with additional partitions. I stand by my original opinion to not restrict or cripple the base system. Instead we should further simplify the possibilities for a user to retrieve regular backups, e.g. through archive files generated by openhab-cli backup provided in /srv/openhab2-backup.

[mstormi]

Yeah and next time you need 32G, then 64 etc. And everybody's starting with his initial SD size, and most people "for safety reasons" already start with a bigger one than needed. Believe me this IS an issue already today. We might be in an early phase still but we are yet to encounter this much more often in the future as more and more people will start making use of restores (rather than reinstalls).

EDIT: Granted a too small partition might become a problem if you set it too tight but we know more or less how many MBs we will need. Then let's be generous and maybe double that and we'll be pretty safe to not run into space limitations on package upgrades. Note the part to grow most will be logs and persistence, and those will be moved to the 'expendable' partition. So "/" shouldn't grow substantially. Not fully sure about OH behavior, but we might even get to see some gain in OH server resilience: Whenever the available disk space on / fills up, openHAB and other tools will stop working for sure. But when it's just unable to write to the expendable partition (for logs) or essential (for persistence), it might keep running (with just those 'features' failing).

I'm trying not to restrict or cripple the system in any way. Clearly I want to avoid that, too. The risk of a too small partition setting set aside (and I think it's negligible, see above), why/where do you think my proposal does restrict anything? It's quite the opposite, this will be an enabler for more flexibility: with this change, a user can move to smaller disks easily (not just ever-bigger ones), and it'll be way easier for him to 'outsource' (move) write-intensive or critical data to network or USB attached media, too, simply by mounting these.

[mues-lee]

I wouldn't do this right in the image, but as an optional setting. This way the user can decide if he really wants to use and might have errors because of low disk space.

also i disagree on the "you always need a bigger card", while this is true when you use dd, it is no problem to just use sudo cp -axv /. /mnt/. sudo cp -axv /dev/. /mnt/dev/. This also works for moving the rootfs to a network share or usb drive

to be honest overall i don't understand the need for this concept at all jm2c

[mstormi]

Please re-read my first post more carefully. Remember we discuss about the general case and not your or anybody other's specific situation. We cannot nor want to force anyone to use a specific copy/restore method. There's many people that will use dd or Amanda restore tool or yet some other method such as offline copying of SD cards using windisk32imager. They will suffer from this.

Regarding remaining space on / partition, note the part to grow most will be logs and persistence, and those will no longer be on the "/" partition. So once we introduce the change, "/" usage shouldn't grow substantially any longer.

[mues-lee]

We cannot nor want to force anyone to use

and this is the reason why I think such a profound change to the system should not be part of the image but a optional setting via openhabian-config

[mstormi]

I agree user-chosen option is always preferrable, but it's close to impossible to code into openhabian-config to re-shrink and repartition without destroying anything. See link in my post for steps to do it manually and then try to code that into a script to work on a running UNIX. At least a very risky thing. Would you take the risk of implementing that ? We need to create a / partition of fixed size, possibly a second fixed size one for /essential (we can discuss that one) and - that's the important part - create a partition of the remaining space to mount as /expendable. Raspbian extends the / partition to the available max at some point of its installation routine, so it'll be way easier and safer to change things there. That's why I think we need to do it in the image. Also, this will only ever apply to image users, not to anyone to install openhabian scripts on top of any UNIX. It'll also only ever affect new images thus new users only, and for them it's no change or option, just the way openHABian is setup. It won't affect anyone to upgrade the openhabian-setup script on his running box. If someone truely didn't want to use this (although I don't see any point why not ?), he could still deploy the current (then old) image, it'll continue to work with the most current openhabian-config scripts and packages. So if we still provide that, a user still has got the choice.

[mues-lee]

If someone truely didn't want to use this (although I don't see any point why not ?), he could still deploy the current (then old) image

really you don't see why people might not want to have a non standard folder / partition scheme? and your solution when someone don't want to use your folder / partition scheme they should use an old/outdated image? sorry but that didn't match "we don't want to force anyone to use something" at all

i stick to my opinion that this shouldn't be part of the image as far as i understood this image should help users with minimal linux knowledge to get openHAB running without many problems. and as a user with minimal linux knowledge, they will start to google to find solutions when the encounter problems. There is no need to increase the difficulty for finding a solution with a non standard folder/partition scheme

a separated image with your changes might be an options, for users that really want this changes and also know about the possible downsides and problems that might come with this separated image

jm2c

[mstormi]

Right I don't see why anyone wouldn't want to have a different partitioning scheme as the new one will have advantages beneficial to everyone. Most openhabian users, and all Linux newbies in particular will not care at all if it's the old or new setup as long as it's working. Most will not even notice it. It's their first and only setup and thus "the" standard for them. It is no less of a standard than the old expand-"/"-to-full-disk is. It's just how it looks like in that version. It also doesn't increase difficulty of whatsoever either. Note it also won't change anything for people that used to install their system based on the old image. Note on filesystem level everything will stay the same. Pathes and filenames won't change so even someone to replace an old image based installation by a new image based one will maybe notice the difference but not suffer from any compatibility issues. The probably only exception to this "people don't care" are a couple of UNIX experts like you and me to have a very distinct opinion and preference on how a filesystem setup should look like, but forgive me when I say that this discussion is pretty academic and not worth it and shouldn't stop us from getting improvements out the door that all non-experts will benefit from. If you see any disadvantages in my proposed changes, please raise and let's discuss these on a technical and objective level, but please not based on subjective preference.

Re-Partitioning has to be done in the image or cannot be done at all. It just wouldn't work to do it on the fly via scripting. Doing it inside openhabian-config would include the need for an implementation to be able to move forth and back. Way to difficult and risky. So that's not a deliberate decision we can take based on preference but a must.

A separate image is what I suggest. I just don't see a need to update/maintain the old image as it can still be used, it's not outdated, it'll just require some more time to update the packages to their most current versions at installation time. But I don't mind updating it, too, and keep maintaining it. Time will tell if there's really a need to keep it.

[mstormi]

Seems noone is willing to take on this. Given a not-that-great risk-to-benefit ratio and many other PRs to go, I'll close this for the time being. Will reopen when its time is on ;-)

[timbms]

Hi Is there any update on this thread. I think @mstormi is totally right, his ideas on shrinking make absolutely sense.

[protivizion]

I use Openhabian more than 5 years. It is very nice system and I like it. I had broken SD card more than 5 time at that period. I used SSD to get more stable system, but it not helps. Backup of openhab do not save all my additional components like RTL_433 which is not part of openhabian. dd to mounted NAS was best solution for me but SSD 128G or SD 64G is to big to make backups often. I try ti make recommended RAW of SD to another SD card, but I found, that my expensive Samsung 64G bigger than my cheaper Sandisk SD 64G and all my backups can't be copied too. To make mirroring I have to use twice bigger card 128G. So I have no possibility to use full backup to another card, I need to resize system to use lower space card, and use mirroring. Function of shrink disk space to lower size will be very useful solution. I have colleagues who are using Openhab and have sam problems as me. It is to difficult to shrink disk on another Linux PC, because I have no more linux PCs than openhabian. Please renew question of SD shrinking.

[mstormi]

Backup of openhab do not save all my additional components like RTL_433 which is not part of openhabian.

openHABian contains Amanda which can backup raw devices so for the time being why not use that. It's essentially an automated, managed form of running dd.

Please renew question of SD shrinking.

It's not a question of "want to" but "how to". As you will just need this once a year or even less often, you can do it manually. There's tutorials on the net. But to code this into an universally applicable, reliably working, automated feature is even another one or two leagues above in terms of difficulty. That's essentially why it has not been done to date. I actually tried but failed (so far). See #1261 .