scripts/build_image source images

[AustinSmart]

Hi! Thanks for all the great work here, its a great source for modifying Venus.

I have been trying out scripts/build_image and have run into some issues. I'm hoping you can give detail on your local environment so I can better understand how I can successfully build a debian/venus image.

What are the source images you are using? Are they iso images of working installs? Are they direct downloads of debian/raspbian/venus? Something else? I'm using iso images of an sd card that has had raspbian (buster) and venus (2.90.24) written to it.

Is this script intended to be run from a debian host? (Im running it on ubuntu 22.04)

Most of what I ran into looks to be issues with the source images not being laid out as the script expects, which causes issues with mounting loopback devices partitions, and copying data.

[smurfix]

Umm. This script only works with Debian (NOT Raspbian!) images. Please get a Debian Pi image from https://raspi.debian.net/daily-images/ and try again.

Also, which Pi are you building an image for? I have only tested the script with a Pi 3. I do have a 2 and a 4 lying around and will hopefully get around to that sometime later.

[AustinSmart]

I've tried with raspian and debian images. Im running this for use on a pi4. But I haven't gotten far enough to know if it works yet. The debian image is able to get further along, but it looks like my Venus image may not be correct. I've tried this sources https://updates.victronenergy.com/feeds/venus/candidate/images/beaglebone/ https://updates.victronenergy.com/feeds/venus/release/images/raspberrypi4/

What is your source for the Venus image?

[smurfix]

I was using the Pi2 Venus images, which is of course not quite optimal for a Pi4. I do have a "spare" Pi4 and will test the image builder with it later this week.

[AustinSmart]

I was able to get an image built using these source images, but it does not cleanly write to an SD card. I havent dug into why.

https://updates.victronenergy.com/feeds/venus/candidate/images/raspberrypi4/venus-image-large-raspberrypi4-20220818201051-v2.90~24.rootfs.wic.gz

https://raspi.debian.net/tested/20220808_raspi_4_bookworm.img.xz

[smurfix]

it does not cleanly write to an SD card

Care to post the error messages here, or what do you mean by "not cleanly"?

[IamSierraCharlie]

Hello, I can see this thread when cold - not sure if i should create a new one? Can you run these scripts from Ubuntu or must your desktop be a debian install?

I got this far -

./build_image debianpi/raspi_4_bullseye.img.xz  venus/venus-image-large-raspberrypi4.wic.gz /dev/sde 
+ test 3 '!=' 3
++ id -u
+ test 1000 '!=' 0
+ set +x
You're not root. Restarting with sudo.
+ test 3 '!=' 3
++ id -u
+ test 0 '!=' 0
+ declare -a mpt
+ declare -a lop
+ d=
+ mpt=()
+ lop=()
+ ok=0
+ DEB=1
+ trap cleanup EXIT
++ mktemp -d
+ tmp=/tmp/tmp.puk4BFMoKF
+ ddeb=/tmp/tmp.puk4BFMoKF/deb
+ dvic=/tmp/tmp.puk4BFMoKF/vic
+ dres=/tmp/tmp.puk4BFMoKF/res
+ dfirm=/tmp/tmp.puk4BFMoKF/firm
+ mkdir /tmp/tmp.puk4BFMoKF/deb /tmp/tmp.puk4BFMoKF/vic /tmp/tmp.puk4BFMoKF/res /tmp/tmp.puk4BFMoKF/firm
+ adeb=debianpi/raspi_4_bullseye.img.xz
+ avic=venus/venus-image-large-raspberrypi4.wic.gz
+ ares=/dev/sde
+ test -f debianpi/raspi_4_bullseye.img.xz
+ '[' .xz == .gz ']'
++ losetup -f
+ l=/dev/loop39
+ losetup -P /dev/loop39 debianpi/raspi_4_bullseye.img.xz
losetup: debianpi/raspi_4_bullseye.img.xz: Warning: file does not fit into a 512-byte sector; the end of the file will be ignored.
+ lop+=($l)
+ adeb=/dev/loop39
+ test -d /dev/loop39
+ test -b /dev/loop39
+ test -b /dev/loop39p1
+ p=
+ mount -o ro /dev/loop392 /tmp/tmp.puk4BFMoKF/deb
mount: /tmp/tmp.puk4BFMoKF/deb: special device /dev/loop392 does not exist.
+ cleanup
+ set +x
To evaluate the error, dropping into a shell.
EXIT when done.

I can confirm that there is no /dev/loop392 on my system

[smurfix]

That's supposed to be /dev/loop39p2.

I'll have a look.

[IamSierraCharlie]

Thanks for following up

In the end, I've manually mounted the images as loop devices - which seems to work okay - I have the following: this is my debian image (second partition): /dev/mapper/loop50p2 on /media/scarter/debian type ext4 (rw,relatime)

this is my victron image (second partition): /dev/mapper/loop60p2 on /media/scarter/vic type ext4 (rw,relatime)

This gets me a lot further:

+ test 3 '!=' 3
++ id -u
+ test 1000 '!=' 0
+ set +x
You're not root. Restarting with sudo.
+ test 3 '!=' 3
++ id -u
+ test 0 '!=' 0
+ declare -a mpt
+ declare -a lop
+ d=
+ mpt=()
+ lop=()
+ ok=0
+ DEB=1
+ trap cleanup EXIT
++ mktemp -d
+ tmp=/tmp/tmp.SWIosU91dP
+ ddeb=/tmp/tmp.SWIosU91dP/deb
+ dvic=/tmp/tmp.SWIosU91dP/vic
+ dres=/tmp/tmp.SWIosU91dP/res
+ dfirm=/tmp/tmp.SWIosU91dP/firm
+ mkdir /tmp/tmp.SWIosU91dP/deb /tmp/tmp.SWIosU91dP/vic /tmp/tmp.SWIosU91dP/res /tmp/tmp.SWIosU91dP/firm
+ adeb=/media/scarter/debian
+ avic=/media/scarter/vic
+ ares=/dev/sde
+ test -f /media/scarter/debian
+ test -d /media/scarter/debian
++ realpath /media/scarter/debian
+ ln -s /media/scarter/debian /tmp/tmp.SWIosU91dP/deb
+ test -f /media/scarter/vic
+ test -d /media/scarter/vic
++ realpath /media/scarter/debian
+ ln -s /media/scarter/debian /tmp/tmp.SWIosU91dP/vic
+ test -e /dev/sde
+ test -f /dev/sde
+ test -d /dev/sde
+ test -b /dev/sde
+ test -b /dev/sdep2
+ test -b /dev/sde2
++ blockdev --getsz /dev/sde
+ sz=31116288
+ sz0=2048
+ sz1=1024000
+ sz2=30090240
+ cat
+ sfdisk /dev/sde
Checking that no-one is using this disk right now ... OK

Disk /dev/sde: 14.86 GiB, 15931539456 bytes, 31116288 sectors
Disk model: MassStorageClass
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xf66f0719

Old situation:

>>> Script header accepted.
>>> Script header accepted.
>>> Created a new DOS disklabel with disk identifier 0xc8801bb5.
/dev/sde1: Created a new partition 1 of type 'W95 FAT32 (LBA)' and of size 500 MiB.
Partition #1 contains a vfat signature.
/dev/sde2: Created a new partition 2 of type 'Linux' and of size 14.4 GiB.
/dev/sde3: Done.

New situation:
Disklabel type: dos
Disk identifier: 0xc8801bb5

Device     Boot   Start      End  Sectors  Size Id Type
/dev/sde1  *       2048  1026047  1024000  500M  c W95 FAT32 (LBA)
/dev/sde2       1026048 31116287 30090240 14.4G 83 Linux

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
+ udevadm settle
+ test -b /dev/sdep1
+ p=
++ blkid /dev/sde1 -o value --match-tag TYPE
+ typ=vfat
+ '[' vfat = '' ']'
++ blkid /dev/sde2 -o value --match-tag TYPE
+ typ=
+ '[' '' = '' ']'
+ mkfs.btrfs -L RASPIROOT /dev/sde2
btrfs-progs v5.4.1 
See http://btrfs.wiki.kernel.org for more information.

Label:              RASPIROOT
UUID:               c35599d1-937f-437e-b321-6e04f2936d7d
Node size:          16384
Sector size:        4096
Filesystem size:    14.35GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP             256.00MiB
  System:           DUP               8.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1    14.35GiB  /dev/sde2

+ mount -o subvol=/ /dev/sde2 /tmp/tmp.SWIosU91dP/res
+ mpt+=($dres)
+ mount /dev/sde1 /tmp/tmp.SWIosU91dP/firm
+ mpt+=($dfirm)
+ test -d /tmp/tmp.SWIosU91dP/res/debian
+ btrfs subvolume create /tmp/tmp.SWIosU91dP/res/debian
Create subvolume '/tmp/tmp.SWIosU91dP/res/debian'
+ test -d /tmp/tmp.SWIosU91dP/res/venus
+ btrfs subvolume create /tmp/tmp.SWIosU91dP/res/venus
Create subvolume '/tmp/tmp.SWIosU91dP/res/venus'
+ test -d /tmp/tmp.SWIosU91dP/res/data
+ btrfs subvolume create /tmp/tmp.SWIosU91dP/res/data
Create subvolume '/tmp/tmp.SWIosU91dP/res/data'
+ btrfs subvolume set-default /tmp/tmp.SWIosU91dP/res/venus
+ RS='rsync -a --partial -x --numeric-ids'
+ rsync -a --partial -x --numeric-ids /tmp/tmp.SWIosU91dP/vic/. /tmp/tmp.SWIosU91dP/res/venus/.
+ rsync -a --partial -x --numeric-ids /tmp/tmp.SWIosU91dP/vic/data/. /tmp/tmp.SWIosU91dP/res/data/.
rsync: change_dir "/tmp/tmp.SWIosU91dP/vic/data" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]
+ cleanup
+ set +x
To evaluate the error, dropping into a shell.
EXIT when done.
root@Jalyn:/# 

Just to cover off what I'm trying to achieve - I saw your post about using this script to build a working raspberry pi 4 image.

https://community.victronenergy.com/questions/151583/raspberry-pi-4b-1.html

I have a working version I d/l from somewhere, but I cant update it via the usual channel. As a result, I thought I might try my hand at building my own - or at least getting the working kernel from Debian to go with the Venus image.

[IamSierraCharlie]

I got further again.
I fired up PyCharm and was able to provide absolute paths to the images.
Images had to be decompressed. I can now write an image and can run it up in my raspberry pi - it all boots, however not detected on Victron connect and only shows the following when I try to log in at the IP

I'd say your script runs fine now that I have my head around how it works. I'll try a few different Victron images to see which one works. My first attempt was https://updates.victronenergy.com/feeds/venus/release/images/raspberrypi4/venus-image-large-raspberrypi4.wic.gz

[smurfix]

Images had to be decompressed.

Yeah, I should have noted that one in the docs.

[IamSierraCharlie]

So I've tried a few different combos of debian / venus but am not having any luck. They all seem to build fine with your script. The first issue I ran into is that the root file system appears to be mounted as read only. I've fixed that in /etc/fstab Secondly it appears that the victron software doesnt actually run. The OS boots to a command prompt and that is all. Have you run this up on a new Pi 4 as yet ? Did you run into this issue?

[smurfix]

There is no such thing as a new "Pi 4" these days, but I did manage to get a CM4 with baseboard and will try that this week.

[Dukat-Gul]

Hi @smurfix . Thank you for this cool work effort. Great concept. My attempts to put it into action resulted in the following feedback for you.

mkfs.btrfs: command not found Requires - apt install btrfs-progs

using venus-image-raspberrypi4-20220715124217-v2.89.rootfs.wic.gz and 20230102_raspi_4_bullseye.img.xz both decompressed xz -d 20230102_raspi_4_bullseye.img.xz and gzip -d venus-image-raspberrypi4-20220715124217-v2.89.rootfs.wic.gz

~/moat-victron/scripts# ./build_image 20230102_raspi_4_bullseye.img venus-image-raspberrypi4-20220715124217-v2.89.rootfs.wic /dev/sda           + test 3 '!=' 3
++ id -u
+ test 0 '!=' 0
+ declare -a mpt
+ declare -a lop
+ d=
+ mpt=()
+ lop=()
+ ok=0
+ DEB=1
+ trap cleanup EXIT
++ mktemp -d
+ tmp=/tmp/tmp.jI4AHIwCiZ
+ ddeb=/tmp/tmp.jI4AHIwCiZ/deb
+ dvic=/tmp/tmp.jI4AHIwCiZ/vic
+ dres=/tmp/tmp.jI4AHIwCiZ/res
+ dfirm=/tmp/tmp.jI4AHIwCiZ/firm
+ mkdir /tmp/tmp.jI4AHIwCiZ/deb /tmp/tmp.jI4AHIwCiZ/vic /tmp/tmp.jI4AHIwCiZ/res /tmp/tmp.jI4AHIwCiZ/firm
+ adeb=20230102_raspi_4_bullseye.img
+ avic=venus-image-raspberrypi4-20220715124217-v2.89.rootfs.wic
+ ares=/dev/sda
+ test -f 20230102_raspi_4_bullseye.img
+ '[' img == .gz ']'
++ losetup -f
+ l=/dev/loop0
+ losetup -P /dev/loop0 20230102_raspi_4_bullseye.img
+ lop+=($l)
+ adeb=/dev/loop0
+ test -d /dev/loop0
+ test -b /dev/loop0
+ test -b /dev/loop0p1
+ p=p
+ mount -o ro /dev/loop0p2 /tmp/tmp.jI4AHIwCiZ/deb
+ mpt+=($ddeb)
+ mount -o ro /dev/loop0p1 /tmp/tmp.jI4AHIwCiZ/deb/boot/firmware
+ mpt+=($ddeb/boot/firmware)
+ test -f venus-image-raspberrypi4-20220715124217-v2.89.rootfs.wic
+ '[' wic == .gz ']'
++ losetup -f
+ l=/dev/loop1
+ losetup -P /dev/loop1 venus-image-raspberrypi4-20220715124217-v2.89.rootfs.wic
+ lop+=($l)
+ avic=/dev/loop1
+ test -d /dev/loop1
+ test -b /dev/loop1
+ test -b /dev/loop1p1
+ p=p
+ mount -o ro /dev/loop1p2 /tmp/tmp.jI4AHIwCiZ/vic
+ mpt+=($dvic)
+ mount -o ro /dev/loop1p1 /tmp/tmp.jI4AHIwCiZ/vic/u-boot
+ mpt+=($dvic/u-boot)
+ mount -o ro /dev/loop1p4 /tmp/tmp.jI4AHIwCiZ/vic/data
mount: /tmp/tmp.jI4AHIwCiZ/vic/data: special device /dev/loop1p4 does not exist.
+ test -e /dev/sda
+ test -f /dev/sda
+ test -d /dev/sda
+ test -b /dev/sda
+ test -b /dev/sdap2
+ test -b /dev/sda2
+ p=
++ blkid /dev/sda1 -o value --match-tag TYPE
+ typ=vfat
+ '[' vfat = '' ']'
++ blkid /dev/sda2 -o value --match-tag TYPE
+ typ=
+ '[' '' = '' ']'
+ mkfs.btrfs -L RASPIROOT /dev/sda2
btrfs-progs v5.10.1
See http://btrfs.wiki.kernel.org for more information.

Label:              RASPIROOT
UUID:               fdec51f9-da90-4287-89c5-810355ceab75
Node size:          16384
Sector size:        4096
Filesystem size:    29.30GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP             256.00MiB
  System:           DUP               8.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Runtime features:
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1    29.30GiB  /dev/sda2

+ mount -o subvol=/ /dev/sda2 /tmp/tmp.jI4AHIwCiZ/res
+ mpt+=($dres)
+ mount /dev/sda1 /tmp/tmp.jI4AHIwCiZ/firm
+ mpt+=($dfirm)
+ test -d /tmp/tmp.jI4AHIwCiZ/res/debian
+ btrfs subvolume create /tmp/tmp.jI4AHIwCiZ/res/debian
Create subvolume '/tmp/tmp.jI4AHIwCiZ/res/debian'
+ test -d /tmp/tmp.jI4AHIwCiZ/res/venus
+ btrfs subvolume create /tmp/tmp.jI4AHIwCiZ/res/venus
Create subvolume '/tmp/tmp.jI4AHIwCiZ/res/venus'
+ test -d /tmp/tmp.jI4AHIwCiZ/res/data
+ btrfs subvolume create /tmp/tmp.jI4AHIwCiZ/res/data
Create subvolume '/tmp/tmp.jI4AHIwCiZ/res/data'
+ btrfs subvolume set-default /tmp/tmp.jI4AHIwCiZ/res/venus
+ RS='rsync -a --partial -x --numeric-ids'
+ rsync -a --partial -x --numeric-ids /tmp/tmp.jI4AHIwCiZ/vic/. /tmp/tmp.jI4AHIwCiZ/res/venus/.
+ rsync -a --partial -x --numeric-ids /tmp/tmp.jI4AHIwCiZ/vic/data/. /tmp/tmp.jI4AHIwCiZ/res/data/.
+ rsync -a --partial -x --numeric-ids /tmp/tmp.jI4AHIwCiZ/deb/boot/firmware/. /tmp/tmp.jI4AHIwCiZ/firm/.
+ '[' 1 = 0 ']'
+ rsync -a --partial -x --numeric-ids /tmp/tmp.jI4AHIwCiZ/deb/. /tmp/tmp.jI4AHIwCiZ/res/debian/.
+ cd /tmp/tmp.jI4AHIwCiZ/res
+ '[' 1 = 0 ']'
+ cp -au --reflink debian/lib/modules/5.10.0-20-arm64 venus/lib/modules/
+ cd venus
+ sed -i -e '/^\/dev\/root\s/d' -e '/\s\/u-boot\s/d' -e '/\s\/data\s/d' etc/fstab
+ grep -qs RASPIROOT etc/fstab
+ cat
+ cd /tmp/tmp.jI4AHIwCiZ/deb/lib
+ for f in ld-*.so.*
+ test -L /tmp/tmp.jI4AHIwCiZ/res/venus/lib/ld-linux-aarch64.so.1
+ ln -s /opt/debian/lib/ld-linux-aarch64.so.1 /tmp/tmp.jI4AHIwCiZ/res/venus/lib/
+ cd /tmp/tmp.jI4AHIwCiZ/res/venus
+ mkdir -p opt/debian boot/firmware
+ fgrep -xqs i2c_dev etc/modules
+ echo i2c_dev
+ cd /tmp/tmp.jI4AHIwCiZ/firm
+ grep -qs '^dtparam=i2c_arm=on' config.txt
+ echo dtparam=i2c_arm=on
+ cd /tmp/tmp.jI4AHIwCiZ/res/debian
+ grep -qs '^dtparam=i2c_arm=on' etc/default/raspi-firmware-custom
+ echo dtparam=i2c_arm=on
+ set +x
All done, unmounting. Good luck!

Worked really well - from first impressions. SD card booted, Was able to access the web interface and enable SSH after the standard victron steps to change access level to superuser.

I wanted to follow your guide/installation steps to run the setupdeb.sh but I couldnt find it. Perhaps I didnt look in the right place(s).

So I ran a opkg update and then opkg install git so I could re clone the repo to the unit running the combined images. Then I ran the setupdeb.sh successfully.

:~/moat-victron/scripts# ./setupdeb.sh
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 http://security.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:4 http://security.debian.org/debian-security bullseye-security/main arm64 Packages [236 kB]
Get:5 http://security.debian.org/debian-security bullseye-security/main Translation-en [158 kB]
Get:6 http://security.debian.org/debian-security bullseye-security/non-free Translation-en [344 B]
Get:7 http://deb.debian.org/debian bullseye/main arm64 Packages [8071 kB]
Get:8 http://deb.debian.org/debian bullseye/main Translation-en [6240 kB]
Get:9 http://deb.debian.org/debian bullseye/non-free arm64 Packages [73.0 kB]
Get:10 http://deb.debian.org/debian bullseye/non-free Translation-en [92.2 kB]
Get:11 http://deb.debian.org/debian bullseye-updates/main arm64 Packages [12.0 kB]
Get:12 http://deb.debian.org/debian bullseye-updates/main Translation-en [7929 B]
Fetched 15.1 MB in 8s (1835 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  liblzo2-2
Suggested packages:
  duperemove
The following NEW packages will be installed:
  btrfs-progs liblzo2-2
0 upgraded, 2 newly installed, 0 to remove and 25 not upgraded.
Need to get 729 kB of archives.
After this operation, 4277 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bullseye/main arm64 liblzo2-2 arm64 2.10-2 [51.8 kB]
Get:2 http://deb.debian.org/debian bullseye/main arm64 btrfs-progs arm64 5.10.1-2 [677 kB]
Fetched 729 kB in 0s (4238 kB/s)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LC_CTYPE = "en_US.utf8",
        LANG = (unset)
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
Selecting previously unselected package liblzo2-2:arm64.
(Reading database ... 18228 files and directories currently installed.)
Preparing to unpack .../liblzo2-2_2.10-2_arm64.deb ...
Unpacking liblzo2-2:arm64 (2.10-2) ...
Selecting previously unselected package btrfs-progs.
Preparing to unpack .../btrfs-progs_5.10.1-2_arm64.deb ...
Unpacking btrfs-progs (5.10.1-2) ...
Setting up liblzo2-2:arm64 (2.10-2) ...
Setting up btrfs-progs (5.10.1-2) ...
Processing triggers for initramfs-tools (0.140) ...
update-initramfs: Generating /boot/initrd.img-5.10.0-20-arm64
Processing triggers for libc-bin (2.31-13+deb11u5) ...

After rebooting, and logging in, I needed to remember to chroot /opt/debian before I treated it as per normal debian installation.

I managed to perform an apt update and then apt upgrade understandably, this within the chroot environment updates files into the /boot which isnt reflected on the separate mmcblk1p1 partition where the actual boot files reside.

cp /opt/debian/boot/initrd.img-5.10.0-23-arm64 /run/media/mmcblk1p1
cp /opt/debian/boot/System.map-5.10.0-23-arm64 /run/media/mmcblk1p1
cp /opt/debian/boot/vmlinuz-5.10.0-23-arm64 /run/media/mmcblk1p1

edited vi /run/media/mmcblk1p1/config.txt to reflect the newer file versions for the next boot. and I referenced the libs - so it could boot correctly.

root@raspberrypi4:/lib/modules# ln -s /opt/debian/lib/modules/5.10.0-23-arm64 5.10.0-23-arm64

On the next boot I could login to the Web interface or SSH. Pings were responding and the HDMI console (which I enabled with some

hdmi_force_hotplug=1
hdmi_group=1
hdmi_mode=31

entries in the config.txt.

I have tried the VenusOS web interface update to a newer version and its not happy.

Thanks again.... Really amazing capability.