Epic: Set up 2nd OS for HASS server

[130s]

Need arose in https://github.com/130s/hut_10sqft/issues/641#issuecomment-1007631680, https://github.com/130s/hut_10sqft/issues/572#issuecomment-1017538801

1. [x] https://github.com/130s/hut_10sqft/issues/698
2. [x] OS installed on an SD card.
   1. [x] RPi4 initialized, installed. https://github.com/130s/hut_10sqft/issues/705
3. [x] HASS installed. https://www.home-assistant.io/installation/linux#install-home-assistant-container Install HASS' Docker image.
4. [x] HASS config backed up remotely https://github.com/130s/essay_in_idleness/issues/43
5. [x] Z-Wave gateway installed and running. https://github.com/130s/hut_10sqft/issues/706
6. Synch the configuration from old HASS server https://github.com/130s/30y-130s_life/issues/1531
   1. [x] Z-Wave device grouping on HASS
   2. [x] Areas, zones
   3. [ ] Synch HASS integration
      • [x] Ring.com -> Camera images nicely showing! Win for newer version of software.
      • [x] Ikea blinds -> https://github.com/130s/30y-130s_life/issues/1505
      • [x] AirQ https://github.com/130s/hut_10sqft/issues/677#issuecomment-1288527056
      • [x] Levoit https://github.com/130s/hut_10sqft/issues/683
      • [ ] Google Nest
      • Else?
7. [x] Zigbee gateway running https://github.com/130s/hut_10sqft/issues/656
8. [x] Include some Zigbee lights https://github.com/130s/hut_10sqft/issues/644#issuecomment-1286457013

[130s]

RPi OS prep on SD card https://wiki.debian.org/RaspberryPiImages

$ export RPI_MODEL=4
$ export SD_CARD=/dev/mmcblk0p1
$ export DEBIAN_RELEASE=bullseye
$ wget https://raspi.debian.net/daily/raspi_${RPI_MODEL}_${DEBIAN_RELEASE}.img.xz
$ wget https://raspi.debian.net/daily/raspi_${RPI_MODEL}_${DEBIAN_RELEASE}.img.xz.sha256
  raspi_4_bullseye.img.xz: OK
$ sha256sum -c raspi_${RPI_MODEL}_${DEBIAN_RELEASE}.img.xz.sha256
$ xzcat raspi_${RPI_MODEL}_${DEBIAN_RELEASE}.img.xz | dd of=${SD_CARD} bs=64k oflag=dsync status=progress

[130s]

Thought xzcat was successful but it failed.

$ sudo xzcat raspi_${RPI_MODEL}_${DEBIAN_RELEASE}.img.xz | dd of=${SD_CARD} bs=64k oflag=dsync status=progress
dd: failed to open '/dev/mmcblk0p1': Permission denied

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.5G     0  7.5G   0% /dev
:
/dev/mmcblk0p1   30G   32K   30G   1% /media/n130s/3737-6362
$ df -h /dev/mmcblk0p1 
Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p1   30G   32K   30G   1% /media/n130s/3737-6362
$ df /dev/mmcblk0p1 
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/mmcblk0p1  31154688    32  31154656   1% /media/n130s/3737-6362
$ ll /media/n130s/3737-6362/
total 36K
drwxr-x---+ 4 root  root  4.0K Jan 20 11:01 ../
drwxr-xr-x  2 n130s n130s  32K Dec 31  1969 ./

[130s]

https://askubuntu.com/a/620857/24203 gave a hint. dd needs sudo as well. So the working command (and the result):

$ sudo xzcat raspi_${RPI_MODEL}_${DEBIAN_RELEASE}.img.xz | sudo dd of=${SD_CARD} bs=64k oflag=dsync status=progress
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 132.952 s, 15.8 MB/s

However I still the destination usage is 0.

$ du -d 1 $SD_CARD 
0       /dev/mmcblk0p1

[130s]

Removed sudo from xzcat but still the same.

$ xzcat raspi_${RPI_MODEL}_${DEBIAN_RELEASE}.img.xz | sudo dd of=${SD_CARD} bs=64k oflag=dsync status=progress
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 132.952 s, 15.8 MB/s

$ sync

$ du -d 1 $SD_CARD 
0       /dev/mmcblk0p1

[130s]

Tried a path one up high. Still the same...

$ export SD_CARD=/dev/mmcblk0
$ ll raspi_${RPI_MODEL}_${DEBIAN_RELEASE}.img.xz && xzcat raspi_${RPI_MODEL}_${DEBIAN_RELEASE}.img.xz | sudo dd of=${SD_CARD} bs=64k oflag=dsync status=progress
-rw-rw-r-- 1 n130s n130s 378M Jan 20 03:51 raspi_4_bullseye.img.xz
2087247872 bytes (2.1 GB, 1.9 GiB) copied, 135 s, 15.5 MB/s
29990+3977 records in
29990+3977 records out
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 135.963 s, 15.4 MB/s
$ sync
$ du -d 1 $SD_CARD 
0       /dev/mmcblk0
$ du -d 1 /dev/mmcblk0
mmcblk0    mmcblk0p1  mmcblk0p2  
$ du -d 1 /dev/mmcblk0p1
0       /dev/mmcblk0p1
$ du -d 1 /dev/mmcblk0p2
0       /dev/mmcblk0p2

But in https://github.com/130s/hut_10sqft/issues/332#issuecomment-457923686 the same command for a different file still showed the size of mmcblk0* to be 0. So maybe I'm good???

[130s]

Wait, it's not even mounted. I don't understand fully but can dd copy without the destination mounted?

$ lsblk|grep mmcb
mmcblk0     179:0    0  29.7G  0 disk 
├─mmcblk0p1 179:1    0   396M  0 part 
└─mmcblk0p2 179:2    0   1.6G  0 part 

[130s]

Indeed, I mounted the device in question and I do see these files. So no problem.

$ sudo mkdir -p /media/noodler/hamster && sudo mount /dev/mmcblk0p1 /media/noodler/hamster
$ ll /media/noodler/hamster/
total 75M
drwxr-xr-x 3 root root 4.0K Jan 21 08:48 ../
-rwxr-xr-x 1 root root  26K Jan 20 03:15 bcm2711-rpi-4-b.dtb*
-rwxr-xr-x 1 root root  14K Jan 20 03:15 bcm2837-rpi-3-a-plus.dtb*
-rwxr-xr-x 1 root root  14K Jan 20 03:15 bcm2837-rpi-3-b.dtb*
-rwxr-xr-x 1 root root  15K Jan 20 03:15 bcm2837-rpi-3-b-plus.dtb*
-rwxr-xr-x 1 root root  14K Jan 20 03:15 bcm2837-rpi-cm3-io3.dtb*
-rwxr-xr-x 1 root root   98 Jan 20 03:15 cmdline.txt*
-rwxr-xr-x 1 root root 1.2K Jan 20 03:15 sysconf.txt*
-rwxr-xr-x 1 root root  278 Jan 20 03:15 config.txt*
-rwxr-xr-x 1 root root  27M Jan 20 03:15 initrd.img-5.10.0-10-arm64*
-rwxr-xr-x 1 root root  27M Jan 20 03:15 vmlinuz-5.10.0-10-arm64*
-rwxr-xr-x 1 root root  52K Jan 20 03:10 bootcode.bin*
-rwxr-xr-x 1 root root 3.2K Jan 20 03:10 fixup4cd.dat*
-rwxr-xr-x 1 root root 5.4K Jan 20 03:10 fixup4.dat*
-rwxr-xr-x 1 root root 8.3K Jan 20 03:10 fixup4db.dat*
-rwxr-xr-x 1 root root 8.3K Jan 20 03:10 fixup4x.dat*
-rwxr-xr-x 1 root root 3.2K Jan 20 03:10 fixup_cd.dat*
-rwxr-xr-x 1 root root 7.2K Jan 20 03:10 fixup.dat*
-rwxr-xr-x 1 root root  11K Jan 20 03:10 fixup_db.dat*
-rwxr-xr-x 1 root root  11K Jan 20 03:10 fixup_x.dat*
-rwxr-xr-x 1 root root 775K Jan 20 03:10 start4cd.elf*
-rwxr-xr-x 1 root root 3.6M Jan 20 03:10 start4db.elf*
-rwxr-xr-x 1 root root 2.2M Jan 20 03:10 start4.elf*
-rwxr-xr-x 1 root root 2.9M Jan 20 03:10 start4x.elf*
-rwxr-xr-x 1 root root 775K Jan 20 03:10 start_cd.elf*
-rwxr-xr-x 1 root root 4.6M Jan 20 03:10 start_db.elf*
-rwxr-xr-x 1 root root 2.9M Jan 20 03:10 start.elf*
-rwxr-xr-x 1 root root 3.6M Jan 20 03:10 start_x.elf*
drwxr-xr-x 2 root root  16K Dec 31  1969 ./

I didn't check the file system format. Turned out vfat.

$ df -T /media/noodler/hamster/
Filesystem     Type 1K-blocks  Used Available Use% Mounted on
/dev/mmcblk0p1 vfat    405264 75816    329448  19% /media/noodler/hamster

So next up is to investigate the file system type RPi4 takes and re-format if needed.

[130s]

Getting RPiOS following https://github.com/130s/hut_10sqft/issues/698#issuecomment-1214137651.

https://www.home-assistant.io/installation/raspberrypi

• Micro SD Card. Ideally get one that is Application Class 2 as they handle small I/O much more consistently than cards not optimized to host applications. A 32 GB or bigger card is recommended.

From https://www.amazon.com/gp/product/B005LFT3QG/ref=ox_sc_act_title_2?smid=AK0X39W9935WX&th=1 via https://github.com/130s/hut_10sqft/issues/606#issuecomment-971162121, the app class is unclear that my 32GB SD card...

Going fancy this time in the interest of time (i.e. avoiding CLI operations like I had been doing above, as I learned less no matter how many times I've gone through of them...) Balena Etcher installation on Ubuntu https://github.com/balena-io/etcher?d_id=4836be44-b4c7-4e96-a5dd-2ab763c87c2eR#debian-and-ubuntu-based-package-repository-gnulinux-x86x64

curl -1sLf \
   'https://dl.cloudsmith.io/public/balena/etcher/setup.deb.sh' \
   | sudo -E bash
sudo apt-get update
sudo apt-get install balena-etcher-electron

Is my RPi4 64/32bit? Looks like the one I bought in 2020 is 64bit.

Starting balena etcher. CLI command couldn't be figured out. Running from "app" gui, I didn't see a GUI popped up, ended up killing all processes 1 by 1. Found out command:

/opt/balenaEtcher/balena-etcher-electron &

I managed to find RPi4 peripherals incl. micro HDMI <--> HDMI cable, but I don't see HDMI output coming through on the external monitor. Then http://X.X.X.X:8123 shows HASS setup in progress. This made me realize that I chose the option to install full OS via HASS's method, which I'm afraid gives less control on OS than I want.

The option I should choose might be Install Home Assistant Container.

RPi OS installation

https://www.raspberrypi.com/software/

$ sudo dpkg -i imager_1.7.2_amd64.deb 
Selecting previously unselected package rpi-imager.
(Reading database ... 462568 files and directories currently installed.)
Preparing to unpack imager_1.7.2_amd64.deb ...
Unpacking rpi-imager (1.7.2) ...
Setting up rpi-imager (1.7.2) ...
Processing triggers for desktop-file-utils (0.24-1ubuntu3) ...
Processing triggers for gnome-menus (3.36.0-1ubuntu1) ...
Processing triggers for bamfdaemon (0.5.3+18.04.20180207.2-0ubuntu2) ...
Rebuilding /usr/share/applications/bamf-2.index...
Processing triggers for mime-support (3.64ubuntu1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...

Meanwhile balena etcher is gone.

$ sudo apt purge balena-etcher-electron 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  balena-etcher-electron*
0 upgraded, 0 newly installed, 1 to remove and 15 not upgraded.
After this operation, 228 MB disk space will be freed.
Do you want to continue? [Y/n] 
(Reading database ... 462575 files and directories currently installed.)
Removing balena-etcher-electron (1.7.9) ...
Processing triggers for desktop-file-utils (0.24-1ubuntu3) ...
Processing triggers for mime-support (3.64ubuntu1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for gnome-menus (3.36.0-1ubuntu1) ...
Processing triggers for bamfdaemon (0.5.3+18.04.20180207.2-0ubuntu2) ...
Rebuilding /usr/share/applications/bamf-2.index...
(Reading database ... 462417 files and directories currently installed.)
Purging configuration files for balena-etcher-electron (1.7.9) ...

On rpi-imager GUI chose RPi OS 64b lite desktop (0.7GB).

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
:
mmcblk0     179:0    0  29.7G  0 disk 
├─mmcblk0p1 179:1    0   256M  0 part 
└─mmcblk0p2 179:2    0   3.7G  0 part 

[130s]

Migrate from a HASS server to a new HASS instance

TLDR; Failed so re-doing configs.

What https://www.home-assistant.io/integrations/backup/ shows doesn't seem to be existent on my 2021.12 HASS. So, alternatively I'm looking into if copy-paste-ing yaml files would do.

• Looks like these files represents configs.

  $ ll .storage/core.*
  core.analytics        core.area_registry    core.config_entries   core.device_registry  core.entity_registry  core.restore_state

  .storage/core.area_registry

  ``` { "version": 1, "minor_version": 1, "key": "core.area_registry", "data": { "areas": [ { "name": "living_room", "id": "living_room", "picture": null }, { "name": "Kitchen", "id": "kitchen", "picture": null }, { "name": "bedroom_master", "id": "bedroom", "picture": null }, { "name": "1F_Office", "id": "1f_office", "picture": null }, { "name": "bedroom_guest", "id": "guest_room", "picture": null }, { "name": "door_front", "id": "door_front", "picture": null }, { "name": "bedroom_slider", "id": "bedroom_leila", "picture": null }, { "name": "2F_media_room", "id": "2f_media_room", "picture": null }, { "name": "dining_formal", "id": "dining_formal", "picture": null }, { "name": "bathroom_1F", "id": "1f_bathroom", "picture": null }, { "name": "garage-1", "id": "garage_1", "picture": null }, { "name": "backyard", "id": "backyard", "picture": null }, { "name": "2F_hallway", "id": "2f_hallway", "picture": null }, { "name": "bedroom_nomad", "id": "bedroom_nomad", "picture": null }, { "name": "garage-2", "id": "garage_2", "picture": null }, { "name": "2F_bathroom_master", "id": "2f_bathroom_master", "picture": null } ] } } ``` ``` ## System Health version | core-2021.12.3 -- | -- installation_type | Home Assistant Core dev | false hassio | false docker | false user | root virtualenv | true python_version | 3.9.8 os_name | Linux os_version | 5.4.0-1071-raspi arch | armv7l timezone | America/New_York

  Home Assistant Cloud

  logged_in | true -- | -- subscription_expiration | October 25, 2022 at 8:00 PM relayer_connected | true remote_enabled | true remote_connected | true alexa_enabled | true google_enabled | true remote_server | us-east-1-2.ui.nabu.casa can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok

  Lovelace

  dashboards | 1 -- | -- resources | 0 mode | auto-gen ```
• reddit from 4 years ago.

  Backup all your config files (including the db file if used, i personally use Maria SQL for the recorder), install hass.io on the new server, copy all the files, restart, done. If you had any kind of ssl certificate with access from outside your network make sure to either backup the certificates, or re-create them.

Evaluating whether the config files include system generated values, which I think I should refrain from copy-paste-ing.

• core.analytics: There's uuid that I can't tell what it is.
• core.area_registry: Seems good, no system generated values.
• core.config_entries: Seems to contain Entities. entry_id, uuid seem sys-gen-ed. Maybe worth trying though? WTF it stores login creds for some of the entities' target (e.g. router).
• core.device_registry: Worth trying?
• core.entity_registry: Worth trying?
• core.restore_state: Not sure if we need this.

Ok, stopping the target services.

• HASS: Looks like I've started HASS Docker container by manually running docker run like https://www.home-assistant.io/installation/raspberrypi#platform-installation.

  docker run -d \
   --name homeassistant \
   --privileged \
   --restart=unless-stopped \
   -e TZ=MY_TIME_ZONE \
   -v /PATH_TO_YOUR_CONFIG:/config \
   --network=host \
   ghcr.io/home-assistant/home-assistant:stable

  So, stopping also manually:

  docker stop homeassistant && docker system prune

• Z-Wave JS:

  cd %PATH_DCOMPOSE%
  docker-compose down

For some reasons the configs copied from the backup didn't seem to be read (e.g. zones were not read) so abandoning those. It's ok just have to start over configuring. Not much done anyway.

[130s]

• [ ] Include some Zigbee lights

I've included 2 of the lightbulbs without any issue. Initially I thought I'd need a Sengled's proprietary Zigbee hub, but I didn't need it (at least for the functionality available via Zigbee).

• For one of the bulb, I triggered inclusion after I turned off - turned on the light on the wall button.
• FOr 2nd one, as soon as I started inclusion on Hub, it was included.

[130s]

• Google Nest

Not urgent, will be figured out.