home-assistant / operating-system

:beginner: Home Assistant Operating System
Apache License 2.0
5.01k stars 981 forks source link

HassOS on SSD #164

Closed hugodelaroza closed 4 years ago

hugodelaroza commented 6 years ago

Is it possible ton install HassOS on an SSD disk on a Raspberry Pi 3?

Does anybody knows how?

pvizeli commented 6 years ago

We want add a feature that it is allow to offload the data partition (they will be not RO) to a external drive. But that come with next version.

hugodelaroza commented 6 years ago

Sorry, I can't understand you. Is there an addon that will allow this? or the next version of HASSOS will?

mbo18 commented 6 years ago

As I understand, we will have to wait for the next HassOS release. Regarding SSD/eMMC, will this also work for the Asus tinker board S? It has a 16Gb eMMC soldered. To burn an image on it you just have to connect it on a computer with a USB cable and use etcher. So no offload needed.

tdolder commented 6 years ago

Boot from SSD would be great feature.

ghost commented 6 years ago

With the new update, there is amention of the odroid eMMC commit. I wonder if that means we can use SSD on pi3 bas well.

rbray89 commented 6 years ago

@tj31moll Unfortunately there would be some effort required to allow non-SD card booting.

mbo18 commented 6 years ago

What about the Asus tinker board S eMMC? Got one, maybe I can help?

shyne99 commented 6 years ago

@rbray89 Can you please develop on this bit ? I was thinking of diving into this matter at some point, SD card are just unreliable. Can you please develop on what is blocking for the moment ? Thank you

rbray89 commented 6 years ago

https://github.com/home-assistant/hassos/commit/960e11dc4f7f32fa3c4937a432ee79d0fe8f1686 was merged to dev. It includes a change for the tinkerboard that should provide the potential for booting off of eMMC, however due to not having the device, the change has yet to be verified.

USB booting is unfortunately usually out of the question, as there exists a chicken-in-the-egg scenario where the USB disk drivers have to be provided at boot.

mrphyslaw commented 6 years ago

So you are saying that no usb boot support will EVER be added to HassOS? This is a huge reliability complaint from the community.

rbray89 commented 6 years ago

@murphyslaw05 It's not possible to boot directly from USB for most embedded devices. We'll attempt to boot from eMMC and USB when available, but it isn't always possible. You should note that there ARE plans to allow users to move the data partition onto USB as @pvizeli mentioned above.

You should also be aware that good quality SD cards often have no issues over long-running installs. (read the tips here: https://www.raspberrypi.org/documentation/installation/sd-cards.md)

If this continues to be an issue for you, I'd suggest moving to a platform where we support eMMC booting, or an efi x86 install.

shyne99 commented 6 years ago

Hi, Thank you for your feedback. As a user of SD cards on SBC in a professional way. I can assure you that SSD are way more reliable than SD card. SD card are the less reliable way to go for system partition, and good quality SD card only mitigate the issue, or postpone the inevitable partition corruption .

I think that your way of only putting boot partition o. SD card and lay the rest in SSD is a good solution. And this is actually a mandatory feature for a home automation system (again with reliability in mind)

mrphyslaw commented 6 years ago

@murphyslaw05 It's not possible to boot directly from USB for most embedded devices. We'll attempt to boot from eMMC and USB when available, but it isn't always possible. You should note that there ARE plans to allow users to move the data partition onto USB as @pvizeli mentioned above.

You should also be aware that good quality SD cards often have no issues over long-running installs. (read the tips here: https://www.raspberrypi.org/documentation/installation/sd-cards.md)

If this continues to be an issue for you, I'd suggest moving to a platform where we support eMMC booting, or an efi x86 install.

Thanks for the suggestions. I have a good quality card, over sized power supply, along with a ups, and do not have corruption issues, but many people do. However there is no viable way to have any sort of write intensive functionality. I am not familiar enough with the OS structure to know much about the limitations, I believe that some sort of version of external drive option was available with ResinOS. This is why myself and others are asking.

thomashup commented 5 years ago

@murphyslaw05 It's not possible to boot directly from USB for most embedded devices. We'll attempt to boot from eMMC and USB when available, but it isn't always possible. You should note that there ARE plans to allow users to move the data partition onto USB as @pvizeli mentioned above.

You should also be aware that good quality SD cards often have no issues over long-running installs. (read the tips here: https://www.raspberrypi.org/documentation/installation/sd-cards.md)

If this continues to be an issue for you, I'd suggest moving to a platform where we support eMMC booting, or an efi x86 install.

Hi.

Sorry for my possible lack of knowledge, but for the RPi 3B+ it's possible to boot directly from USB when using Eg. Raspbian - I know hassio is based on another system, but isn't it somehow possible to make a Hassio release for RPi 3B+ that makes it possible to boot from USB like eg. Raspbian?

And by the way, you are doing a fantastic job. :)

/Thomas.

cholek3 commented 5 years ago

@murphyslaw05 It's not possible to boot directly from USB for most embedded devices. We'll attempt to boot from eMMC and USB when available, but it isn't always possible. You should note that there ARE plans to allow users to move the data partition onto USB as @pvizeli mentioned above. You should also be aware that good quality SD cards often have no issues over long-running installs. (read the tips here: https://www.raspberrypi.org/documentation/installation/sd-cards.md) If this continues to be an issue for you, I'd suggest moving to a platform where we support eMMC booting, or an efi x86 install.

Hi.

Sorry for my possible lack of knowledge, but for the RPi 3B+ it's possible to boot directly from USB when using Eg. Raspbian - I know hassio is based on another system, but isn't it somehow possible to make a Hassio release for RPi 3B+ that makes it possible to boot from USB like eg. Raspbian?

And by the way, you are doing a fantastic job. :)

/Thomas.

I think you are getting confused with terminology here. There are two versions of Hass.io. One is based on new system (HassOS) and other one is based on old system (ResinOS). You cannot boot from Hass.io based on HassOS, but you can boot from Hass.io based on ResinOS.

taanczos commented 5 years ago

Is there any progress with this question? I really would like to use Hassio (hassos version) from msata with my rpi3b+ :( I already lost 2 sd cards (sandisk ultras ...), now hassio is on a sandisk extreme plus, but I don't feel safe :) Regards Tamás

ghost commented 5 years ago

Is any chance to add this feature?

szurr commented 5 years ago

Hi Maybe there is a way to put only db files to usb?

mbo18 commented 5 years ago

@berg seems to be working on it: https://github.com/home-assistant/hassos/pull/318

ghost commented 5 years ago

Waiting to see result of this one https://github.com/home-assistant/hassos/pull/318 seems to be merged already into Dev

Ayo91 commented 5 years ago

Is there any progress with this question? I really would like to use Hassio (hassos version) from msata with my rpi3b+ :( I already lost 2 sd cards (sandisk ultras ...), now hassio is on a sandisk extreme plus, but I don't feel safe :) Regards Tamás

I have the same problem... I lost 2 sd cards too.. And they were a good brand. After that, I create automatic backups and upload to Dropbox. We need usb boot :(

nsotnikov commented 5 years ago

+1 an usb boot option would be very great, I have unused usb ssd that waits to run as an hassos hard drive

Ayo91 commented 5 years ago

Yes, my Pi has broken 2 Sandisk SD card. After some months they were corrupted.... We need this funcionality...

I have been researching and it seems a solution is install HA in other way around:https://www.home-assistant.io/docs/installation/raspberry-pi/

Any feedback?

jvanderneutstulen commented 5 years ago

The main issue the data partition which gets written to a lot. The PR from @berg should be enough to move the data partition to an external drive. Basically it's two steps:

  1. Create ext4 partition with label hassos-data on the external drive.
  2. Rename the partition hassos-data on the sdcard to something else.

With the RPI 3 you can (permanently) enable usb boot mode, but this will probably require changes in the uboot configuration, so a different image for usb and mmc. As an alternative it may be posssible to create a small usb-boot uboot image, so the bootloader lives on the sdcard and the other stuff is on the external disk. This principle can work with other targets as well.

ghost commented 5 years ago

@Ayo91 You are correct that many other methods exist. This is specifically for those that prefer the HassIO method using the self contained HassOS image. 🙂

@jvanderneutstulen The boot loader is an interesting idea as it would rarely need to be anything other than a read only device.
I wonder if maybe that is an option @Berg could work with rather than fully moving it all to a USB device. You are right that the MMC and USB would likely need to be separate images, unless HassOS can be accept a txt config file with an option that could be toggled between USB and MMC boot modes

Ayo91 commented 5 years ago

I wonder if maybe that is an option @berg could work with rather than fully moving it all to a USB device. You are right that the MMC and USB would likely need to be separate images, unless HassOS can be accept a txt config file with an option that could be toggled between USB and MMC boot modes

It will be awesome support that.

Ayo91 commented 5 years ago

The main issue the data partition which gets written to a lot. The PR from @berg should be enough to move the data partition to an external drive. Basically it's two steps:

  1. Create ext4 partition with label hassos-data on the external drive.
  2. Rename the partition hassos-data on the sdcard to something else.

Has someone tested that? Because my HA is in live, its a monitoring system and an alarm and I dont want to break anything and stop the system. Should I migrate the content of the SD hassoss-data to the
new partition that I will create?

With the RPI 3 you can (permanently) enable usb boot mode, but this will probably require changes in the uboot configuration, so a different image for usb and mmc. As an alternative it may be posssible to create a small usb-boot uboot image, so the bootloader lives on the sdcard and the other stuff is on the external disk. This principle can work with other targets as well.

Some instructions for do that? small usb-boot uboot image and the rest in the SSD.

Thanks.

tboyce021 commented 5 years ago

The main issue the data partition which gets written to a lot. The PR from @berg should be enough to move the data partition to an external drive. Basically it's two steps:

  1. Create ext4 partition with label hassos-data on the external drive.
  2. Rename the partition hassos-data on the sdcard to something else.

Has someone tested that? Because my HA is in live, its a monitoring system and an alarm and I dont want to break anything and stop the system. Should I migrate the content of the SD hassoss-data to the new partition that I will create?

I just tried this on my NUC (running HassOS VMDK on Proxmox) and it seems to have worked. I booted to an Ubuntu live USB and used gparted to rename/relabel the old partition to hassos-data-orig and create the new partition on a separate disk. Then I copied the data over using

sudo dd if=/dev/sda8 of=/dev/sdb1 bs=4096 conv=notrunc,noerror,sync

(update to use your paths)

That seems to have changed the label on the new partition to match the old one (hassos-data-orig), so I used gparted again to relabel the new partition to hassos-data.

Rebooted HassOS, briefly saw a message about it resizing the disk/partition, and can now see the new size with df -h /mnt/data.

You may be able to do this on a running system, but this was easier for me since I could ensure everything was shut down and not writing to the disk when I copied the contents.

ghost commented 5 years ago

@tboyce021 Did you try the standard image as well as the virtual disk image? In theory they should be the same, but the virtual disk image may have some tweaks in it. 🙂

tboyce021 commented 5 years ago

I only tried the VMDK. However, after just updating to 2.10, my HassOS VM won't boot anymore. Not sure if it's related to moving the data partition or not though.

Ayo91 commented 5 years ago

I deleted HassOs. Bad performance ever. I installed a Ubuntu server in a computer with a consumption of 25W and I'm very happy. Everything works nice and very stable. No more SD broken.

iLLiac4 commented 5 years ago

Hi. Is this already available in the latest image? I have tried what tboyce021 wrote but it does not boot. I there any official statement how to achive that on hassio?

iLLiac4 commented 5 years ago

Hi. To answer my own question :) Yes it is possible to migrate data from sd card to ssd.

I have tested few scenarios and they do work.

  1. Clean install:
  1. Migration:
    • Create ext4 on SSD with gparted.
    • Clone hassos-data from your production sd card to SSD with command: sudo dd if=/dev/sda8 of=/dev/sdb bs=4096 conv=notrunc,noerror,sync
    • After clonning label sd card hassos-data to hassos-data-old.
    • On SSD label of data partition has to be hassos-data.
    • To extend the data partition of newly cloned ssd select from gparted menu Partition/Check and it will extend it.
    • Insert sd card into rpi and newly created ssd and it will boot.
tboyce021 commented 5 years ago

do not use of=/dev/sdb1 but sdb

Awesome find. For me, /dev/sdb1 would work fine initially, but subsequent boots would fail. With /dev/sdb, reboots seem to be working again.

Also, you don't need to create a new ext4 partition this way. It just uses the whole disk so you can skip that step.

FYI to others, you can add status=progress to the end of the dd command to get progress updates while it runs (assuming your version of dd is new enough).

bbbenji commented 5 years ago

Somethings to note:

iLLiac4 commented 5 years ago

Ehat do you mean with this? DO NOT CLICK UPDATE UNDER HOST SYSTEM CARD

Do you mean that the hassio can not be updated after migration to external ssd?

bbbenji commented 5 years ago

No. I mean that until a new stable build is released, that button will downgrade you to 2.10.

tboyce021 commented 5 years ago

Maybe add (if you're on 3.0+) to clarify. I did that same thing when on a development build once. But I'm currently on 2.10 and it still works fine.

iLLiac4 commented 5 years ago

I was doing those testing on 2.10 image. I did not see a 3.0+ images at all.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

nukefrenzy commented 5 years ago

I have followed iLLiac4's instructions for a new install after my SD card failed for the 2nd time. However, the RPi shows no signs of life except for the red power light. The green activity light never blinks even once. A fresh Hassio install will work just fine. Any ideas?

I used gparted from a USB drive to copy the data folder and followed your steps to the letter. I realize I haven't given many details, but I understand the steps as you wrote them and the SD card and USB drive both look to be correct when I am about to put them in the RPi. I can post more details tonight when I get home from work.

Jpsy commented 5 years ago

However, the RPi shows no signs of life except for the red power light. The green activity light never blinks even once.

I had the exact same effect. Turned out that I had not changed the LABEL of the SD card data partition to hassos-data-old, but the NAME! Gparted allows to change both. You may have to unmount the partition before you can change the label. Unmounting is also possible through Gparted.

ghost commented 5 years ago

When will be implement boot from SSD USB to image HASS.io ?

Misiu commented 5 years ago

Raspberry PI 4 is released and many of us are wondering if this scenario will be possible some day - download HASSIO image, flash it to Pen Drive or SSD and use that instead of SD Card.

ghost commented 5 years ago

I buy RPI4 and wait for it. I also can be beta tester if it is need.

esciara commented 5 years ago

Hi guys,

Just to share my experience of trying @iLLiac4 's steps directly on a RPI. I did it from a brand new install. I guess it can work with an existing setup, but I have not tried, and my knowledge of linux is too limited to vouch for that.

Full procedure

First I followed the Install Hassio doc to create my SD card with balenaEtcher (using the latest hassos_rpi3-2.12.img image) on a mac for a Raspberry Pi 3 Model B+ (32bit as recommended on the doc). Also did the optional part to get a static IP. And setup SSH on port 22222 as explained on the Debugging Hassio dev doc.

Launched my RPI with the "CONFIG" USB stick and a USB drive (target for holding all my Hassio data). All starts normally and as explained in the Debugging Hassio doc, I can SSH from my mac with a ssh root@hassio.local -p 22222 and access the host system use the 'login' command.

From there, lsblk gives me a picture of the partitions:

# lsblk -o name,mountpoint,label,size,uuid
NAME        MOUNTPOINT   LABEL             SIZE UUID
sda                                        1.9G
`-sda1                   CONFIG            1.9G C41D-13F5
sdb                                      465.8G
mmcblk0                                    7.4G
|-mmcblk0p1 /mnt/boot    hassos-boot        32M 25BC-1A86
|-mmcblk0p2              hassos-kernel      24M b3888eda-c63e-472d-a697-388c2a6d74be
|-mmcblk0p3 /                              256M
|-mmcblk0p4                                 24M
|-mmcblk0p5                                256M
|-mmcblk0p6                                  8M
|-mmcblk0p7 /mnt/overlay hassos-overlay     96M c3204cd3-41c0-456d-9305-e01c8e14d9cb
`-mmcblk0p8 /mnt/data    hassos-data       6.7G 73557961-3e93-47cd-ad4f-9c7751ac28d9
zram0       [SWAP]       hassos-zramswap 185.2M 16dad8e5-07b1-49e3-8711-ff4afbe159a6
zram1       /var         hassos-var         32M 1dba1baa-3905-463e-97e7-4ea75a057c40
zram2       /tmp         hassos-tmp         16M 45e52160-1bee-4ae8-a1c8-023cfb87da1a

So in my case, the partition I want to clone is /dev/mmcblk0p8. That is what I do:

# dd if=/dev/mmcblk0p8 of=/dev/sdb bs=4096 conv=notrunc,noerror,sync

At that point, the USB drive has the same hassos-data label as the SD card. I relabel the SD hassos-data to hassos-data-old:

# e2label /dev/mmcblk0p8 hassos-data-old

Checked with lsblk to see if all is fine, and it seems to be: (note that sdb seem to have changed name to sda in the process... pretty sure I did not change USB ports)

# lsblk -o name,mountpoint,label,size,uuid
NAME        MOUNTPOINT   LABEL             SIZE UUID
sda         /mnt/data    hassos-data     465.8G 73557961-3e93-47cd-ad4f-9c7751ac28d9
sdb                                        1.9G
`-sdb1                   CONFIG            1.9G C41D-13F5
mmcblk0                                    7.4G
|-mmcblk0p1 /mnt/boot    hassos-boot        32M 25BC-1A86
|-mmcblk0p2              hassos-kernel      24M b3888eda-c63e-472d-a697-388c2a6d74be
|-mmcblk0p3 /                              256M
|-mmcblk0p4                                 24M
|-mmcblk0p5                                256M
|-mmcblk0p6                                  8M
|-mmcblk0p7 /mnt/overlay hassos-overlay     96M c3204cd3-41c0-456d-9305-e01c8e14d9cb
`-mmcblk0p8              hassos-data-old   6.7G 73557961-3e93-47cd-ad4f-9c7751ac28d9
zram0       [SWAP]       hassos-zramswap 185.2M 9a2aa25f-6450-45db-a89f-5b959e36a9f6
zram1       /var         hassos-var         32M cc6cdd18-9615-43ce-b206-e8ab4232c32d
zram2       /tmp         hassos-tmp         16M 9f729146-0ac1-46e0-8b2d-85a49fded628

Since I am not so good in linux (so I don't really know how to check it smartly), I guess this all means that it has done things as expected. In any case, when I am starting the RPI without the USB drive, it would not work... Whereas it works just fine when the USB drive is plugged.

Finishing notes

Trying to unmount hassos-data-old

Tried to unmount /dev/mmcblk0p8 with fdisk to make sure it was not used, but fdisk talking to me about some complicated hybrid stuff:

# fdisk /dev/mmcblk0

Welcome to fdisk (util-linux 2.32.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

A hybrid GPT was detected. You have to sync the hybrid MBR manually (expert command 'M').

Command (m for help):

That didn't encourage me try further.

Extend the data partition of newly cloned hassos-data on USB drive

This was suggested in @iLLiac4 's steps. Is it really required? If yes, is it possible to do it without GParted?

Any feedback welcome!

pvizeli commented 5 years ago

@esciara look like you know how a Linux system/partition work.

Add a script to /usr/sbin they do that automatically if you set a device as a parameter. That the expander script going wrong, is a bug. Maybe you can fix it?

Like: https://github.com/home-assistant/hassos/blob/dev/buildroot-external/rootfs-overlay/usr/libexec/hassos-expand Use sgdisk for GPT and sfdisk for MBR (on the main harddisk). The new Data partition Harddisk should be all-time GPT because we don't need boot from it.

esciara commented 5 years ago

Sorry for the late reply.

@pvizeli Google is my friend :stuck_out_tongue_winking_eye: . Put what I could find together (with enough understanding of it.

Add a script to /usr/sbin they do that automatically if you set a device as a parameter. That the expander script going wrong, is a bug. Maybe you can fix it?

Not sure I understood. Do you mean there is a bug somewhere in what I wrote? Or are you talking about adding a new feature? Can you clarify, please?

Like: https://github.com/home-assistant/hassos/blob/dev/buildroot-external/rootfs-overlay/usr/libexec/hassos-expand Use sgdisk for GPT and sfdisk for MBR (on the main harddisk). The new Data partition Harddisk should be all-time GPT because we don't need boot from it.

I actually don't plan to touch the partitions... I prefer to do as less modifications as possible to avoid breaking things. Unless you think I should or that it should be done for it to work?

Misiu commented 5 years ago

@pvizeli is something going on to allow booting and running only from SSD/Pendrive? Raspberry Pi3, Pi3B+ and soon Pi4 are/will be able to run from an external drive without the need to use SD card at all. The ideal setup would be to download an image, flash it to SD/SSD/Pendrive plug it in and that's all. No need to manually edit files, move partitions.

Is this even doable for HassOS?

esciara commented 5 years ago

@Misiu : I did quite a bit of research on that, and the conclusion was that HassOS did not allow for that. Don't have the references at hand (please share if you find them), but I remember the alternative was about installing Hassio on raspbian, which has pros and cons, but is definitely more prone to having issues, since HassOS does a lot of work for you...

EDIT: and re-read the comments on this issue, especially the earlier ones... it gives quite a bit of info about why this is complicated...