search

SeedSigner / seedsigner-os

SeedSigner OS | Minimal Raspberry Pi image made for SeedSigner
MIT License
97 stars 25 forks source link

Faster boot times and Reproducible builds #51

Closed newtonick closed 1 year ago

newtonick commented 1 year ago

This pull request has 2 intermingled functional changes. Reducing the time from power on to splash screen and a method for producing deterministic reproducible builds. The main reason these changes are intermingled is because both required the reduction of build dependencies. Specific reproducible steps could be pulled out into its own, but this felt like unnecessary and time consuming work.

Faster boot times don’t need much explanation. I measure the pi0 board booting in 15 seconds instead of 45 in release 0.6.0. The image size is 27262976 bytes total (compared to the previous release at 35652096 bytes, an 8MB of savings).

Having a SeedSigner OS reproducible build allows creation of the image to be compiled and assembled deterministically. This makes it possible for multiple people to compile the same source code into identical binaries. This also means sole trust in SeedSigner the person (or any other SeedSigner contributors) isn’t needed for the image that most people run on thier SeedSigner. This trust can be distributed through others attesting they to can produce the same image byte for byte. Also for those who compile the image themselves, they have a way to verify the image produced is identical to the one released.

Here is a breakdown of configuration and script changes made to accomplish these 2 larger functional goals.

This PR replaces #45 and #34.

Written instructions for how to create reproducible builds: https://gist.github.com/newtonick/2df134a84fd04398bd925a000e979112

Known issues with this PR:

newtonick commented 1 year ago

My reproducible build testing results. All builds done with amd64 architecture. I will update this table as I complete builds.

Build Script Options Image File Name sha256 Output
--pi0 --app-commit-id=9c36f5c seedsigner_os.9c36f5c.pi0.img 29e41c ... 28b036
--pi2 --app-commit-id=9c36f5c seedsigner_os.9c36f5c.pi2.img 2f470d ... 85e6a9
--pi02w --app-commit-id=9c36f5c seedsigner_os.9c36f5c.pi02w.img f99d1b ... be20bc
--pi4 --app-commit-id=9c36f5c seedsigner_os.9c36f5c.pi4.img d29514 ... ee4c5c
kdmukai commented 1 year ago

Success on M1 Pro!!!

Screenshot 2023-07-18 at 12 15 50 AM

2hrs 38min

Note on macOS it's shasum -a 256 [file]

kdmukai commented 1 year ago

So far trying to get this running on Windows10 is a total nightmare (Windows' fault, not @newtonick!).

There are (at least) four possible paths:

Pure Ubuntu via WSL2 (SUCCESS!! 🎉🎉)

Basic steps:

Build time on a Ryzen 5 5600X (6-core / 12-thread cpu): 25 minutes

Notes:

Docker Desktop + PowerShell notes (NOT SUCCESSFUL SO FAR)

For PowerShell the command syntax has to be:

$env:DOCKER_DEFAULT_PLATFORM="linux/amd64"; $env:SS_ARGS="--pi0 --app-commit-id=9c36f5c"; docker compose up --force-recreate --build

UPDATE: Consistently fails here: 20230718_082317

Via the manual build steps

Build the Docker container with:

$env:DOCKER_DEFAULT_PLATFORM="linux/amd64"; docker build -t seedsigner-os-build-host .

Get a bash shell inside the container: but in PowerShell, the command can't be spread across multiple lines and the volume mapping needs the Windows backslash:

docker run --entrypoint bash -ti --name seedsigner-os-build-host-container -v .\opt:/opt -v .\images:/images seedsigner-os-build-host

Once you're inside the container, restore broken symlinks to hashes (otherwise you'll hit ERROR: No hash found for gcc-11.3.0.tar.xz in two places):

mv /opt/buildroot/package/gcc/gcc-initial/gcc-initial.hash /opt/buildroot/package/gcc/gcc-initial/gcc-initial.hash.orig
ln -s /opt/buildroot/package/gcc/gcc.hash /opt/buildroot/package/gcc/gcc-initial/gcc-initial.hash

mv /opt/buildroot/package/gcc/gcc-final/gcc-final.hash /opt/buildroot/package/gcc/gcc-final/gcc-final.hash.orig
ln -s /opt/buildroot/package/gcc/gcc.hash /opt/buildroot/package/gcc/gcc-final/gcc-final.hash

Now run the build script as usual:

./build.sh --pi0 --app-commit-id=9c36f5c

My build was able to finish, however, the hash did not match:

cd6410609b40b1ff8081f0134ee0000a4c82b03d4e06a6079ebdbc04be673ddc

And the build crashed in my SeedSigner when I tried to enable persistent settings. That did not happen with other images that matched the expected hash.

Docker Desktop + Ubuntu (NOT SUCCESSFUL SO FAR)

Basic steps:

Maybe the more sane instructions would be to run the Windows machine as TailsOS for this process. I assume TailsOS runs fine on any Intel/AMD Windows box?

UPDATE: TailsOS + modern nVidia graphics card has no workaround. 20230718_084437

I will edit this comment with more results as I do more testing.

jdlcdl commented 1 year ago

Your gist is such detailed work that I think I learned more about building seedsigner-os in this one document than I have in the past. I love the alternate methods and explanations. I'll note that I recall a discussion like: "Maybe a pre-release and docs for public scrutiny aren't necessary upfront?" and "Maybe not a show-stopper if pi2 and pi4 boards are not supported right away?"... and then you went on to slay them both... all reproducible, all exactly the same file sizes, Wow!

As of your gist at "Revisions=24"...

I've attempted a proof-read from top to bottom: minor typos:

I've followed the instructions, cutting and pasting from your gist, w/o major problems:

Reproduced images for all supported boards appear to match your sha256 hashes; as for this attestation:

$ sha256sum ./images/*
f99d1b27b77264319fbc747c9b3e47c617445002826b248ae2cb8ce9b9be20bc  ./images/seedsigner_os.9c36f5c.pi02w.img
29e41cccf0ca93705901bacd5d605da884f9b8315cfd978910698c2f8e28b036  ./images/seedsigner_os.9c36f5c.pi0.img
2f470dfca0491d97696ccdd9c45104cb1bab0af52a8405e85ef7fbc7cf85e6a9  ./images/seedsigner_os.9c36f5c.pi2.img
d295148e754f48504f559daf50ac92ed8ca2b94b2b22bfbd214cd49146ee4c5c  ./images/seedsigner_os.9c36f5c.pi4.img

The images for the pi0, pi2, and pi4 all boot fast and succeed the "I/O test" (w/ camera_rotation=0 as expected).

kdmukai commented 1 year ago

Notes for running in a LunaNode VPS:

Why do the reproducible build in a Virtual Machine(VM)? The build should be straightforward on a Linux OS but is potentially hopeless on Windows. Windows users are much better off avoiding the Windows complications altogether and building instead on a brand new Ubuntu VM that starts pristine and will be discarded as soon as the build is done.

Why LunaNode?

Create LunaNode instance:

Once it's created:

ssh into VM

From a local terminal: ssh ubuntu@[server's public ip] and use the password from above when prompted.

Install Docker

Once inside the new VM, install Docker via the convenience script: https://docs.docker.com/engine/install/ubuntu/#install-using-the-convenience-script (if the script fails with "Unable to acquire the dpkg frontend lock", just wait a minute or two; the new VM is still running its initial setup steps)

Or if you want more control, you can do a manual install: https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository

Configure Docker permissions

The convenience script recommends a follow-up step to take care of this part:

sudo apt-get install -y uidmap
dockerd-rootless-setuptool.sh install

Or you can do it manually: Steps 1-4: https://docs.docker.com/engine/install/linux-postinstall/

Confirm your Docker setup

docker run hello-world

Build SeedSigner-OS image

(follow @newtonick's notes from there; shouldn't need any modifications now that we're in a totally standard Ubuntu VM)

Build time on a c.2 image: 84m33.988s

Copying the resulting .img off the VM

??

Shouldn't be necessary if the hashes match; you can be confident that downloading the release image from the SeedSigner github repo is identical to building it yourself.

Delete the LunaNode VM instance

In the LunaNode "Virtual Machines" page, click on your instance, and then click the red "Delete" button at the top right and follow the instructions.

gpatkinson commented 1 year ago

all build hashes match on my Ubuntu LTS 20.04 VM. fantastic work, @newtonick

image

newtonick commented 1 year ago
  • "others attesting they to can produce" becomes "others attesting they too can produce"
  • "You should now have an shell session open" becomes "You should now have a shell session open"
  • "You'll need to clone to app repo first" becomes "You'll need to clone the app repo first"
  • "This will produce a produce a list" becomes "This will produce a list"
  • header "install Docker and my recommend configurations" becomes "install Docker and my recommended configurations"

Thank you for these, I have applied these updates to the gist.

Your gist is such detailed work that I think I learned more about building seedsigner-os in this one document than I have in the past. I love the alternate methods and explanations. I'll note that I recall a discussion like: "Maybe a pre-release and docs for public scrutiny aren't necessary upfront?" and "Maybe not a show-stopper if pi2 and pi4 boards are not supported right away?"... and then you went on to slay them both... all reproducible, all exactly the same file sizes, Wow!

As of your gist at "Revisions=24"...

I've attempted a proof-read from top to bottom: minor typos:

  • "others attesting they to can produce" becomes "others attesting they too can produce"
  • "You should now have an shell session open" becomes "You should now have a shell session open"
  • "You'll need to clone to app repo first" becomes "You'll need to clone the app repo first"
  • "This will produce a produce a list" becomes "This will produce a list"
  • header "install Docker and my recommend configurations" becomes "install Docker and my recommended configurations"

I've followed the instructions, cutting and pasting from your gist, w/o major problems:

  • pi0, pi02w and pi2 built w/ "You want more?" instructions (last 2 w/ modified build.sh commandlines).
  • pi4 built from same container, except w/ "Still not satisfied?" instructions and modified make commandline. There was visible error-output in the find...-delete command: find: cannot delete ‘/opt/rootfs-overlay/opt’: Directory not empty ... but it did do it's job -- only src/ remained. Because I had just built for pi2 and output/ was there, I had to rm -rf /output inside the container, then run the two make commands again, modifying the first for pi4. Afterwards, I did cp /output/images/seedsigner-os.img /images/ and renamed it to be more appropriate for my pi4 build.
  • the "Bonus Round" commands all worked. For a quick comparison, the sha256 hash of my rootfs.md5.chk file is e9745263f0d149a50b109c29f230092cdefe74a9e15ef5c33dca663c6fdb5edb. ...but I did get a single line of diff output after comparing. Yours has: d41d8cd98f00b204e9800998ecf8427e ./dev/console while mine does not; all others are same in rootfs for the pi0 image. (might your's have been from a previous build?)
  • each build ran in just under 40m on Ubuntu/x86_64 system w/ 4-core-Xeon and 16GB RAM -- via user-space docker.
  • I did NOT try the docker compose up one-liner (because I don't have docker-compose).

Reproduced images for all supported boards appear to match your sha256 hashes; as for this attestation:

$ sha256sum ./images/*
f99d1b27b77264319fbc747c9b3e47c617445002826b248ae2cb8ce9b9be20bc  ./images/seedsigner_os.9c36f5c.pi02w.img
29e41cccf0ca93705901bacd5d605da884f9b8315cfd978910698c2f8e28b036  ./images/seedsigner_os.9c36f5c.pi0.img
2f470dfca0491d97696ccdd9c45104cb1bab0af52a8405e85ef7fbc7cf85e6a9  ./images/seedsigner_os.9c36f5c.pi2.img
d295148e754f48504f559daf50ac92ed8ca2b94b2b22bfbd214cd49146ee4c5c  ./images/seedsigner_os.9c36f5c.pi4.img

The images for the pi0, pi2, and pi4 all boot fast and succeed the "I/O test" (w/ camera_rotation=0 as expected).

  • pi0 splash at ~15s, main menu at ~26s
  • pi2 splash at ~11s, main menu at ~20s
  • pi4 splash at ~16s, main menu at ~23s
  • I did NOT test the pi02w image.

Great comments and feedback. Thank you for the thorough testing!

jahangir13 commented 1 year ago

Worked in my seedsigner Debian 11 lxd container for the pi0 image (using the One-Liner command): Took about 1 1/2 hours with 2 CPUs assigned (nearly all the time at 99%). Disk space needed for Debian and image build: about 14GB

Installation after deploying the clean Debian11 proxmox container template to the machine): apt update && apt dist-upgrade -y apt install git Installation of docker ('Install using the APT Repository) from docker documenation: https://docs.docker.com/engine/install/debian/ After that post-installation steps might be needed: https://docs.docker.com/engine/install/linux-postinstall/ --> execute one-liner command as root (in my case) to execute the build process

grafik

jahangir13 commented 1 year ago

Worked also in my Windows 10 (WSL2/Ubuntu Terminal) for the pi0 image with the one-liner command (now less than 40 minutes with 6 CPU cores): Installed docker and docker-compose into Ubuntu in Windows: https://docs.docker.com/engine/install/ubuntu/

grafik

kdmukai commented 1 year ago

@jahangir13 maybe my mistake was trying to have Docker Desktop in Windows run the Docker service for WSL2.

I'll see if I can go back and stay purely in Ubuntu/WSL2.

jahangir13 commented 1 year ago

@jahangir13 maybe my mistake was trying to have Docker Desktop in Windows run the Docker service for WSL2.

I'll see if I can go back and stay purely in Ubuntu/WSL2.

Yes, exactly. I just opened the Ubuntu Terminal (I did not start Docker Desktop at all). Then installed docker stuff in the Terminal (so in Ubuntu). I now also give up with Powershell for now,

kdmukai commented 1 year ago

I also discovered that when Docker Desktop had the WSL2 integration enabled, it was slamming my loud HDD zzt-zzt-zzt-zzt nonstop. For hours. Likely forever.

As soon as I stopped Docker Desktop, all those read/writes stopped. When I disabled the WSL2 integration and Docker Desktop restarted, it left my HDD in peace.

kdmukai commented 1 year ago

Another confirmation, via twitter (yes, weight it accordingly)

https://twitter.com/4moonsettler/status/1682057433298894852

kdmukai commented 1 year ago

Big f'n ACK.

I'm satisfied that it works on enough platforms. Documentation and step-by-step guidance (esp for Windows) still needed, but recommend that be done in follow up PRs (and can be submitted by more contributors to distribute the workload).

jdlcdl commented 1 year ago

I've finally succeeded with the one-liner method, for pi2-dev. sha256sum looks like:

b7f738035f4794b0f8b50afd07a25d980345868690864830ea3867ab05dfca00  ./images/seedsigner_os.9c36f5c.pi2-dev.img

ACK

newtonick commented 1 year ago

On the VPS topic, just another example of building on VPS. Here is a Linode VPS script I have used to build a test SeedSigner OS image. It's 100% command line through Linodes command line client. I'm sharing it because it's an example of building the SeedSigner OS image.

Create Linode VPS

export LINODE_TMP_ROOT_PASSWD=$(head /dev/urandom | LC_ALL=C tr -dc A-Za-z0-9 | head -c 20)
echo "Linode Root Password: ${LINODE_TMP_ROOT_PASSWD}"
export LINODE_TMP_SSH_AUTH_KEY=$(cat ~/.ssh/id_ed25519.pub)

linode-cli linodes create \
 --type g6-dedicated-8 \
 --region us-east \
 --image linode/debian11 \
 --root_pass ${LINODE_TMP_ROOT_PASSWD} \
 --authorized_keys "${LINODE_TMP_SSH_AUTH_KEY}" \
 --label seedsigner-os

export LINODE_SEEDSIGNER_OS_IP=$(linode-cli linodes ls --label seedsigner-os --json | jq -r '.[0].ipv4[0]')

Connect via SSH to VPS once server is provisioned and starts sshd service

ssh -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no" root@${LINODE_SEEDSIGNER_OS_IP}

Install Docker, Docker Compose and then Build the SeedSigner OS Image on VPS

apt update -qq
apt install  -qq -y ca-certificates curl gnupg

# Add Docker’s official GPG key:
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
chmod a+r /etc/apt/keyrings/docker.gpg

# Use the following command to set up the repository:
echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker Engine and Docker Compose.
apt update -qq
sudo apt-get install -y docker-ce docker-ce-cli docker-compose-plugin

# Build SeedSigner OS Image
git clone --recursive https://github.com/SeedSigner/seedsigner-os.git
cd seedsigner-os
git fetch origin pull/51/head:pr_51
git checkout pr_51
git submodule update
DOCKER_DEFAULT_PLATFORM="linux/amd64" SS_ARGS="--pi0 --app-commit-id=9c36f5c" docker compose up --force-recreate --build
uptime
exit

Delete Linode VPS when done

export LINODE_SEEDSIGNER_OS_ID=$(linode-cli linodes ls --label seedsigner-os --json | jq -r '.[0].id')
linode-cli linodes delete ${LINODE_SEEDSIGNER_OS_ID}

Output, last few lines for reference

42 minutes of uptime to build in this one example

seedsigner-os-build-images-1  | /opt/buildroot
seedsigner-os-build-images-1  | 29e41cccf0ca93705901bacd5d605da884f9b8315cfd978910698c2f8e28b036  /opt/../images/seedsigner_os.9c36f5c.pi0.img
seedsigner-os-build-images-1 exited with code 0
root@localhost:~/seedsigner-os# uptime
 21:06:08 up 42 min,  1 users,  load average: 3.77, 4.89, 4.32
root@localhost:~/seedsigner-os# exit
logout
Connection to 50.116.57.212 closed.
user@machine ~ % export LINODE_SEEDSIGNER_OS_ID=$(linode-cli linodes ls --label seedsigner-os --json | jq -r '.[0].id')
linode-cli linodes delete ${LINODE_SEEDSIGNER_OS_ID}
hugoender commented 1 year ago

This is amazing! If only all open source projects had this type of thorough documentation for their repro builds. Amazing work @newtonick

Marc-Gee commented 1 year ago

Windows Success using docker desktop !! (via the single line command ) The pi02w hash matches ! f99d1b27b77264319fbc747c9b3e47c617445002826b248ae2cb8ce9b9be20bc

image
hugoender commented 1 year ago

As I was running the following command ./build.sh --pi0 --app-commit-id=9c36f5c I noticed in the very numerous text that was scrolling through the terminal that there was a connection to http://mirrors.tripadvisor.com/gnu/mpc/mpc-1.2.1.tar.gz in there (see last lines of screenshot below). Why is there a connection to tripadvisor.com?!

Screenshot from 2023-07-21 21-43-24

jahangir13 commented 1 year ago

http://mirrors.tripadvisor.com/gnu/mpc/mpc-1.2.1.tar.gz

They provide a GNU mirror server (as you also could do). These can be seen in this list: https://www.gnu.org/prep/ftp.en.html

And depending on your location and network state it selects one from the list: grafik

At least this is my understanding ;)

hugoender commented 1 year ago

http://mirrors.tripadvisor.com/gnu/mpc/mpc-1.2.1.tar.gz

They provide a GNU mirror server (as you also could do). These can be seen in this list: https://www.gnu.org/prep/ftp.en.html

And depending on your location and network state it selects one from the list.

At least this is my understanding ;)

That is VERY interesting. Never thought a company like TripAdvisor would provide a GNU mirror. Is there any benefit to them to do that or would it just be something they do because they believe in the project?

hugoender commented 1 year ago

Feedback on this PR. The instructions are great but there is one small detail missing for those of us that are not very familiar with Docker: how do you pull or transfer the image that's created after ./build.sh --pi0 --app-commit-id=9c36f5c from the Docker container to the host? Is this even necessary or can you flash from within Docker container?

EDIT: Nevermind, I saw that this is talked about in the first section. Therefore, my recommendation would be to provide a link in the You Want More section that takes you to the top section where it explains this.

newtonick commented 1 year ago

I am merging this PR knowing there are some small pi4 issues and lots of opportunity for improvements. This PR accomplishes what it was set out to do even if imperfectly. I'm looking forward to PRs following it to make everything more solidified. I'm happy with the testing and review many stepped in to perform. Thank you!