gitbls / sdm

Raspberry Pi SD Card Image Manager
MIT License
462 stars 48 forks source link

General questions around SDM #76

Closed mattie47 closed 1 year ago

mattie47 commented 1 year ago

Hi there,

Firstly, this application looks awesome, and is extremely well documented.

I've spent some time reading through the docs (and watched the youtube demo), and I've got a couple questions before I try this myself.

1. How "easily" are you able to do version control of configuration for an image?

One thing I'm a bit unclear on is how someone may be able to make a custom image based on RaspiOS and store all the things necessary minus the image itself in git?

I see there's examples where you go directly into the image using functionality such as --explore, however I imagine any changes here are specific to said image?

I guess the --explore is more of an option, rather than the main way to do config...

Use case, or what I want to do:

  1. Download latest RaspiOS 32bit 1a. git clone several repos (from private repos accessible on server/build agent), and copy the repos into a directory to be later moved to the user's home directory
  2. Configure basic config e.g. ssh, wifi, hostname, user, enable vnc etc
  3. Enable /boot/config.txt custom configuration enable numerous overlays, i2c, etc
  4. Move custom data from step 1a to user's home directory
  5. Perform other custom configuration e.g. move numerous pre-defined config files into the user's .profile/ directory etc.

Basically I've got a bunch of custom stuff, or stuff I'd like to be quite obvious in a repo for someone else trying to use SDM to build a customer image. I get the impression I can do most of the above, however the impression I get is that most stuff is through command switches (of which they are extensive - nice!), however I guess as someone who in general understands where stuff goes in linux, can I simply do something like:

mv repo-folder/root/etc/* sdm/root/etc/*

I realize with that statement I'm horribly over-simplifying things, but I guess I'm trying to understand whether SDM can be configured to install all the relevant applications, and then I copy existing config files, or append to numerous configs through the OS?

Or am I not thinking far ahead enough?

For reference I've spent some time trying to evaluate different options, whether that be https://github.com/RPi-Distro/pi-gen or simply modify the Pi's firstboot.sh script to point to a bunch of other scripts and copy necessary files, and install applications.

2. How easily can I script building of an image?

Perhaps this is the tl;dr question of all of the above stated in question 1.

3. How well is the WSL support currently working?

I imagine the answer will be "try it".

I guess a semi related question - Does SDM work in Docker? Again this links back to my scriptibility question,

Perhaps for CICD purposes?

Thanks in advance,

Matt

gitbls commented 1 year ago

Great that you find sdm interesting! I'll take a shot at answering your questions, if I miss anything or didn't explain something clearly enough, let's iterate as needed.

Version control in an IMG. Certainly doable! I'm not a git wizard, but if you make the contents that you use in a particular IMG a git version or git branch, and tie that version/branch to that IMG in some way, you'll be able to retrieve it at any time, right?

Taking that a step further, if you want the contents of that version/branch actually IN the IMG, you can easily do that by copying the files (or entire tree) from your host system into the IMG.

Here's an outline of how I would handle what you've described. I'd encourage you to have a good look at https://github.com/gitbls/sdm/blob/master/Docs/Example-Plugin.md to see what a plugin looks like. This example is based on my personal plugin.

sdm is very different from pi-gen. You can read about some of the differences here. I should put this into the docs at some point.

How easily can I script building an IMG? This is pretty simple. See https://github.com/gitbls/sdm/blob/master/Docs/Example-Commands.md for an example. You can do this as simply or as complex as you want.

How well is WSL support working? It works well for doing IMG customization. At the current time you can't burn an SSD/SD Card directly from WSL, this is a WSL limitation. I saw something about a long-winded set of steps that might enable burning, but it's nothing I would consider documenting for sdm. That said, if you figure it out, I'd be interested in hearing how you did it (step-by-step of course 😄) I've only used it with a Debian WSL2 system, but it should work with any Debian derivative system, such as Ubuntu.

One note on WSL: On Windows X64 systems, sdm will run Phase 1 and the post-install Phase in a chroot using qemu emulation. On my X64-10th Gen system it runs at about Pi4 speed.

Does sdm work in Docker? No clue. Nobody's asked about that. You can certainly use sdm to install Docker into an IMG so that the system comes up with it ready to rock, but I've not tried running sdm inside Docker.

Hope this helps get you jump-started. More questions? Fire away!

1stcall commented 1 year ago

sdm works well using wsl. I have found the best way to deal with flashing the image to SD card, is to use the --burnfile option instead of --burn to create an .img file, then use the Raspberry Pi Imager app to flash the image onto a SD card.

The customised image file is also suitable for flashing the eMMC Flash on a Compute module using rpiboot.

If you wish to burn the SD card directly from wsl, the process is as follows.

  1. Start an elevated PowerShell prompt.

  2. Insert the SD card.

  3. Enter the PowerShell command 'Get-WmiObject -query "SELECT * from Win32_DiskDrive"' to list the available drives. The output should be simular to :-

    Partitions : 3 DeviceID : \\.\PHYSICALDRIVE1 Model : BC511 NVMe SK hynix 256GB Size : 256052966400 Caption : BC511 NVMe SK hynix 256GB

    Partitions : 2 DeviceID : \\.\PHYSICALDRIVE2 Model : SDHC Card Size : 15924142080 Caption : SDHC Card

    Partitions : 3 DeviceID : \\.\PHYSICALDRIVE0 Model : Vaseky V800/240G Size : 240054796800 Caption : Vaseky V800/240G

  4. From the output you should be able to identify the DeviceID of the SD card. From the example above, it would be \\.\PHYSICALDRIVE2'.

  5. Mount the card into wsl using the PowerShell command 'wsl --mount DeviceID --bare'. This then makes SD card available to you wsl distribution under /dev/sdX. the actual device name may vary depending upon devices on your system.

  6. Use the --burn option as usual.

  7. I'm not sure this is necessary, but I then run the commands 'sudo sync && sudo eject /dev/sdX' to flush the write buffer and make the device safe for removal.

  8. Run the PowerShell command 'wsl --unmount DeviceID' to remove the device from wsl.

  9. Safely Remove and Eject the device form the Windows Taskbar.

I hope this helps.

gitbls commented 1 year ago

Thanks @1stcall for writing this up! I'll be giving it a go soon, and will update the sdm docs if I'm successful. 👍

mattie47 commented 1 year ago

@gitbls just wanted to say thanks for your quick and useful feedback.

I've been playing around with SDM over this week and it really does seem to fit my need :-)

I'm trying to make some notes as I go, which I'm hoping I can use to give some useful feedback/docs at some point.

Just juggling this around other work, so watch this space :)

Thanks,

Matt

gitbls commented 1 year ago

Thanks for the positive feedback! I'm always interested in feedback on my tools, so bring it on! Also, if you have more questions as you proceed down the path to total sdm enlightenment, please feel free to post them.

mattie47 commented 1 year ago

Hi @gitbls ,

A couple questions.

I've been trying to use the following:

sudo sdm \
--customize \
--wpa /usr/local/sdm/vhs-files/etc/wpa_supplicant/wpa_supplicant.conf \
--keymap us \
--locale en_NZ.UTF-8 \
--timezone Pacific/Auckland \
--wifi-country NZ \
--restart \
--password-pi password1\! \
--password-root password1\! \
--autologin \
--apps "dos2unix wireguard minicom can-utils xterm screen neofetch" \
--bootset boot_splash=0,i2c=0,spi=0,boot_order=B2 \
--dtoverlay gpio-shutdown,gpio_pin=18 \
--dtoverlay mcp251xfd,spi0-0,interrupt=23 \
--dtoverlay mcp251xfd,spi0-1,interrupt=25 \
/usr/local/sdm/images/2023-02-21-raspios-bullseye-armhf.img \
/

When I try to run this, I get a few "odd" things I can't quite figure out.

~~1. Is it expected on initial boot to get the "Welcome to Raspberry Pi desktop!" screen? I'm assuming not, but I can't quite see what I may have wrong to cause this.

  1. The above screen disappears after around 30 seconds, and then I seem to be able to boot straight into raspiOS, however I'm a bit stuck on whether I have a locale issue, or simply not getting the password stuff right.
  2. First desktop screen is the "configure locale" however it's correctly picked up my NZ locale stuff (minus ticking the US keyboard layout). Note that I was trying to set locale statically after some issues with --L10n (which may or may not be due to wsl, so I'm just trying to rule things out at this point).
  3. Next it's asking me to create the raspberry Pi user account. I don't know why I'm getting this, as the pi user is correctly configured with the password1 password specified above.
  4. Do you happen to know how keymapping works? Sorry this is probably more of a generic linux question, but I was trying to figure out how I specify both XKBMODEL="pc105" and XKBLAYOUT="us assuming I need to specify the pc105 type?~~

Can you see anything logical for why I might be having issues?

Figured it out - If you're using an image like RaspiOS / Raspbian and you don't want the "Welcome to Raspberry Pi desktop" screens to pop up, then you need to add --disable piwiz to your --configure flags.

I had assumed this first time "wizard" would auto-disappear given we're configuring everything manually. For reference, you don't see the first time wizard if you set things manually in recent versions of Raspberry Pi Imager which is why I didn't realise it was a config flag option in itself to run, rather than verifying "all the things" were configured.

Several related questions:

As you can see, I basically have had to delete the output image, and recreate based on a "golden" image each time I want to create an image.

I'm kind of surprised --configure doesn't allow you to specify a source image (for golden configuration), and then your "output" image with your custom config, and what you're then going to burn to SD/USB/TFTP.

Am I just "using it wrong" or is that not an intended workflow?

Finally, I see you can't specify "hostname" in --configure, but there's an assumption you're going to do this in --burn. Right now I'm exploring the image, and manually doing this in /etc/hostname as well as updating /etc/hosts respectively, but I was just curious why --burn would be a requirement for this?

Thanks again (and sorry for the wall of text!),

Matt

gitbls commented 1 year ago

Wow! Lots of questions...here are some answers, in order:

It is my general philosophy for sdm that it not do too many things "automatically". Thus, even though you're specifying all the settings that piwiz wants to ask you about, sdm won't automatically disable piwiz. My take from this is that perhaps the sdm docs should more liberally mention --disable piwiz in the descriptions of --keymap, --locale, etc.

As an aside, if your WSL system is running Debian or Debian derivative (Ubuntu, etc) you can use --L10n to get those settings from the running system.

root password: If all you want is for the user 'pi', or the user you create with --user to be able to sudo, then no, you don't need to set the root password. Only set the root password if you want to be able to LOGIN as root. If you don't want to, for instance, ssh to the pi and LOGIN as root, you don't need the root password. Setting --password-root doesn't set the root password unless --rootpwd is also set, as you noticed. If you use --password-same y the root password will not be set unless --rootpwd is set.

Personally, I use --rootpwd --password-same y --password mypassword. I know that security wonks will chastise me for this. Such is life. This makes it easy for me to pop up an xterm running on my pi logged in as root using https://github.com/gitbls/rxapp and I do this constantly.

Workflow: The general Linux model is that each command does one thing and does it well. Complex functions can be accomplished by combining simple commands in a script. So, no, sdm will not copy an IMG for you. Do that before you do the customize. I do that, just like you, although my "work disk" is NFS-mounted, so I fire off the copy to run remotely on the NFS server since it runs several seconds faster (and I'm rather impatient 🤣)

Hostname. One of the key features of sdm is that a single customized IMG can be used for multiple hosts, with the host identity applied late, e.g., during --burn. Setting a hostname during customize can be overridden during burn, but the original thinking was that everyone would use sdm for burning. Unfortunately, doing an sdm burn with WSL is a bit more complex (but can be done, as @1stcall mentioned above. (I haven't tried this yet, but assuming it works well/reliably I will add it to the sdm docs).

You should try burning from WSL as @1stcall mentioned, because then you can user --expand-root and save a whole reboot of your brand new IMG.

Sounds like you may only have a single Pi. If that's correct, congratulations for getting started with sdm, and your future self will thank your current self when your Pi farm consists of multiple systems. If you already have multiple Pis, I'm sure you'll recognize the value in having a single IMG that can be deployed for multiple Pis. That is the one of the key values of sdm. Every Pi on my network is built from the same IMG, with some getting services added to their disks at burn time (e.g., I only install DHCP, DNS, and VPN servers on the Pis that actually run those services). But the base systems are 100% identical, which makes my life SO much easier.

gitbls commented 1 year ago

A couple of other points: I see that you've opted to put stuff under /usr/local/sdm. While there's no concrete reason to not put things there, I'd encourage you to reconsider this and move them to another directory. For instance, I have a top-level directory named /rpi and I put everything underneath that. My /rpi directory is owned by me, rather than root, which I find to be more convenient.

Also, you can add --csrc /somedir (--csrc /rpi in my case) and then refer to stuff underneath there in your plugins and other scripts as $csrc/rpi/foo (again, using my example).

Also, I noted that you have specified --apps "dos2unix wireguard minicom can-utils xterm screen neofetch", which is fine until the list gets too long. You can add these all to a file (one per line) and then use --apps @thefile on the sdm command line. (this ICYMI in the docs).

gitbls commented 1 year ago

sdm works well using wsl. I have found the best way to deal with flashing the image to SD card, is to use the --burnfile option instead of --burn to create an .img file, then use the Raspberry Pi Imager app to flash the image onto a SD card.

The customised image file is also suitable for flashing the eMMC Flash on a Compute module using rpiboot.

If you wish to burn the SD card directly from wsl, the process is as follows.

  1. Start an elevated PowerShell prompt.
  2. Insert the SD card.
  3. Enter the PowerShell command 'Get-WmiObject -query "SELECT * from Win32_DiskDrive"' to list the available drives. The output should be simular to :- Partitions : 3 DeviceID : \.\PHYSICALDRIVE1 Model : BC511 NVMe SK hynix 256GB Size : 256052966400 Caption : BC511 NVMe SK hynix 256GB Partitions : 2 DeviceID : \.\PHYSICALDRIVE2 Model : SDHC Card Size : 15924142080 Caption : SDHC Card Partitions : 3 DeviceID : \.\PHYSICALDRIVE0 Model : Vaseky V800/240G Size : 240054796800 Caption : Vaseky V800/240G
  4. From the output you should be able to identify the DeviceID of the SD card. From the example above, it would be \.\PHYSICALDRIVE2'.
  5. Mount the card into wsl using the PowerShell command 'wsl --mount DeviceID --bare'. This then makes SD card available to you wsl distribution under /dev/sdX. the actual device name may vary depending upon devices on your system.
  6. Use the --burn option as usual.
  7. I'm not sure this is necessary, but I then run the commands 'sudo sync && sudo eject /dev/sdX' to flush the write buffer and make the device safe for removal.
  8. Run the PowerShell command 'wsl --unmount DeviceID' to remove the device from wsl.
  9. Safely Remove and Eject the device form the Windows Taskbar.

I hope this helps.

@1stcall, I had an opportunity to try this today, but with no success. the wsl --mount command fails with "The system cannot find the drive specified" when I use "\.\PHYSICALDRIVE1". I can see it access the SD card USB reader light flashes, but it fails. If I use single slashes, as you've indicated, I get "The system cannot find the file specified".

The above test was on Win11. Similar errors on Win10. Searching the web tells me that lots of people are having this problem.

So, what is your secret sauce to make this work?

Thanks

mattie47 commented 1 year ago

Thanks again for your feedback and answers @gitbls!

Semi-hoping my many questions/comments will help others in the process of trying this :)

As an aside, if your WSL system is running Debian or Debian derivative (Ubuntu, etc) you can use --L10n to get those settings from the running system.

Yeah that's what I started with, but got confused by piwiz and started trying different things. Thanks though.

Sounds like you may only have a single Pi.

I'm using 1 Pi for now, but I've got about 15 to provision/manage, and admin is a scary thought, so I'm trying to come up with management workflow for now.

Curious if you've settled on any options for administration post-install? E.g. Ansible, Salt etc?

I'm slowly learning that SDM often operates in two mindsets:

If you're not closely paying attention to the examples listed in https://github.com/gitbls/sdm/blob/master/Docs/Example-Commands.md, then you can be forgiven for thinking you've done all you need to, to install each application.

I also grepped on the help output which wasn't immediately obvious poptions was needed.

Perhaps it may be an idea to list in the "help" which things must be set to apply things.

E.g.

 --apps applist      Configure a list of app packages to install or @file with list. Apply using "--poptions apps"
 --xapps applist     Configure a list of X11 packages to install or @file with list. Apply using "--poptions xapps"

I've also noticed when you exit from the mounted terminal (e.g. using --explore) it doesn't return your terminal colours back to "normal". I'm guessing this is an issue with WSL, as opposed to native linux?

Regarding directory location - Thanks.

I had initially:

cd ~/repos/
git clone https://github.com/gitbls/sdm.git
cd sdm

and then tried doing things in there, but ran into some permissions issues (Initially had tried to create an "images" directory within sdm folder). (It's possible I wasn't having a permissions issue, but simply thought that's what it was at the time...)

At that point I noticed in the docs curl -L https://raw.githubusercontent.com/gitbls/sdm/master/EZsdmInstaller | bash and simply moved to doing things in /usr/local/sdm, however at this point everything has simply been "playing around" and trying to understand the tool before moving to proper scripting.

Anyway - I shall continue playing around and see how I go.

Thanks again

gitbls commented 1 year ago

I have done a bit of work with ansible, but never latched onto it as THE solution. This whole area of updating systems once they are running is probably my biggest gaping "problem area", but so far it hasn't bugged me enough to do anything about it 🤷‍♀️

The --poptions apps,xapps was added very early, I think because I wanted an easy way to enable/disable adding the apps. I've thought about removing it, but don't really want to deal with the breakage it would cause to people's sdm scripts, so it is what it is. Thanks for the suggestion on updating the docs/help for this. I put it on the TODO list.

As far as the terminal colors not changing back, I haven't seen that. I use xterms in Debian with WSL, and they work fine. What terminal emulator and distro are you using? (BTW, WSL IS native Linux for the most part, at least WSL2).

Seems like your learning curve is ramping up quickly! Next come the suggestions and wish list 🤣

mattie47 commented 1 year ago

Thanks again.

Rather than leave this thread with no update, I just wanted to mention I'm currently on vacation, and have every intention of picking up from where I left off in about two weeks :-)

gitbls commented 1 year ago

Must be having a great vacation 🤣. Are we going to get back to this at some point? 🤔

mattie47 commented 1 year ago

@gitbls yeah naturally I got back to work with other fires to deal with.

I have every intention of picking this back up. Sorry - as I do want to play around with this!

I use xterms in Debian with WSL, and they work fine. What terminal emulator and distro are you using? (BTW, WSL IS native Linux for the most part, at least WSL2).

WSL2 with standard Ubuntu install. I was using https://github.com/microsoft/terminal as well as the stocked terminal Windows uses by default for WSL2.

Where I was next confused is how to get files transferred into the image. However, after reading https://github.com/gitbls/sdm/blob/master/Docs/Programming-Plugins-and-Custom-Phase-Scripts.md I see there's the ability to cp stuff in via the $SDMPT variable. (is PT "pointer" or something else?)

Also it would be worth linking that doc in the intro/Overview in https://github.com/gitbls/sdm/blob/master/Docs/Custom-Phase-Script.md and https://github.com/gitbls/sdm/blob/master/Docs/Plugins.md

So - would this be a typical workflow?:

What about general file/config manipulation? For example "I need to modify line X in file Y and insert variable abc" as opposed to copying full config files over. I realise this is standard bash scripting, but I was just curious if there's already a good example of things like this already? All good if not.

Thanks again,

Matt

gitbls commented 1 year ago

When $SDMPT popped into my head it stood for "sdm mount point", but best to think of it as the proper way to reference the IMG. It's only defined in Phase 0, but you can use it always (Phase 0, Phase 1, post-install), since in Phase 1 and post-install it's set to "".

Will take a look at your doc recommendations. Thanks for those!

As far as "typical workflow", you understand it correctly.

For general file/config manipulation, I would do that in Phase 1 or post-install, rather than in Phase 0. In Phase 0 you're not running in the IMG, and it's easy to duff it by not using $SDMPT and modify your running system. That's highly unlikely to happen in the later phases, when running in the container (nspawn or chroot).

That said, if you look at the Plugin example you'll note that I do a bit of mucking with files in Phase 0, but it's only because that Phase 0 is involved with setting up the home directory files. Edits to existing files are done in the post-install phase (modifying /etc/fstab and setting the window manager to openbox instead of mutter).

Basically, I try to ensure that I don't screw myself by inadvertantly modifying the running system (I've done that 😲) as well as making sure that the things I want to modify have been installed.

gitbls commented 1 year ago

(finally) linked the Plugins and Custom Phase Script docs from the Programming docs.

gitbls commented 1 year ago

Closing this issue since a long time without input. Please reopen, or start another issue, if needed. Thanks for your inputs!

mattie47 commented 1 year ago

Yeah sorry about that @gitbls .

Thanks again for keeping this open for so long, and the general help you provided.

We got SDM doing what we wanted, but my colleague took over from me and as such I don't have any docs to contribute.

Thanks again for a great tool.

simlu commented 10 months ago

Does sdm work in Docker?

It looks like currently the answer is unfortunately no. But there is hope this might be fixable

https://github.com/gitbls/sdm/issues/155