search

gitbls / sdm

Raspberry Pi SD Card Image Manager
MIT License
469 stars 48 forks source link

Error when running inside Docker #155

Closed simlu closed 9 months ago

simlu commented 11 months ago

First of all, this looks like a great project! As someone who as created much of the code logic here independently, I know how much work was put into this project! Much appreciated!

We are currently using a custom preparation script to modify the pi lite os img pre distribution. We are trying to switch over to using sdm.

However when running sdm inside a docker container, we are getting the following error

* Mount IMG '/user/project/2023-12-11-raspios-bookworm-arm64-lite.img'
mount: /mnt/sdm: special device /dev/loop0p2 does not exist.
? Error mounting IMG '/user/project/2023-12-11-raspios-bookworm-arm64-lite.img'

Is sdm is using loop devices for mounting? It is definitely possible to mount the boot and main partition inside a docker container.

This might be an opportunity to solidify the code used for mounting in sdm. Would appreciate some feedback!

Here is the relevant (working) code from our script:

(code is broken, use this code below instead!)

Expand ```sh # ... echo "Analysing ISO..." isoSectorSize=$(fdisk -l $ISO_FULL_PATH | sed -n -e '/^Sector size/p' | grep -o -E '[0-9]+' | head -1 | sed -e 's/^0\+//') isoBootStart=$(fdisk -l $ISO_FULL_PATH | grep "FAT32" | awk '{print $2}') isoBootEnd=$(fdisk -l $ISO_FULL_PATH | grep "FAT32" | awk '{print $3}') isoBootOffset=$(($isoSectorSize*$isoBootStart)) isoBootSizeLimit=$(($isoSectorSize*$isoBootEnd)) isoMainStart=$(fdisk -l $ISO_FULL_PATH | grep "Linux$" | awk '{print $2}') isoMainEnd=$(fdisk -l $ISO_FULL_PATH | grep "Linux$" | awk '{print $3}') isoMainOffset=$(($isoSectorSize*$isoMainStart)) isoMainSizeLimit=$(($isoSectorSize*$isoMainEnd)) echo "Mounting Boot Partition..." sudo mkdir /mnt/boot sudo mount -v -o offset=$isoBootOffset,sizelimit=$isoBootSizeLimit -t auto $ISO_FULL_PATH /mnt/boot # ... echo "Unmounting..." sudo umount /mnt/boot sudo rm -rf /mnt/boot echo "Mounting Main Partition..." sudo mkdir /mnt/main sudo mount -v -o offset=$isoMainOffset,sizelimit=$isoMainSizeLimit -t ext4 $ISO_FULL_PATH /mnt/main # ... echo "Unmounting..." sudo umount /mnt/main sudo rm -rf /mnt/main # ... ```
simlu commented 11 months ago

Relevant spot in the code is the domount function https://github.com/gitbls/sdm/blob/e810ec5fde930539a4044e9c5f268c4f697e3f0f/sdm-cparse#L1173

gitbls commented 11 months ago

Yes, indeed, sdm uses a loop device to mount the IMG.

Before worrying about loop devices inside docker, here's some homework for you. Please test to see if it's possible to run systemd-nspawn inside docker. If that fails, does chroot work?

And, assuming one of the above two does work, how would sdm determine that it's running inside docker?

simlu commented 11 months ago

Hello and thank you for the quick reply!

1) Any reason that loop devices need to be used? Directly mounting should work just fine everywhere

2) I wasn't able to get systemd-nspawn working, but this whacky test script for chroot ran just fine

Expand ``` mkdir /user/project/jail mkdir /user/project/jail/bin mkdir /user/project/jail/lib64 mkdir /user/project/jail/lib mkdir /user/project/jail/lib/x86_64-linux-gnu cd /user/project/jail cp -v /bin/bash /user/project/jail/bin/bash ldd /bin/bash cp -v /lib/x86_64-linux-gnu/libtinfo.so.6 /user/project/jail/lib/x86_64-linux-gnu/ cp -v /lib/x86_64-linux-gnu/libdl.so.2 /user/project/jail/lib/x86_64-linux-gnu/ cp -v /lib/x86_64-linux-gnu/libc.so.6 /user/project/jail/lib/x86_64-linux-gnu/ cp -v /lib64/ld-linux-x86-64.so.2 /user/project/jail/lib64 chroot /user/project/jail /bin/bash ```

3) Why would sdm need to know that it's running inside of a docker container. Ideally the logic should be the same? But the easiest way to check for it (if this is indeed necessary for some reason) would be to check for existence of /.dockerenv

gitbls commented 11 months ago
  1. I will look into removing the need for a loop device
  2. good that chroot works
  3. sdm would need to know that if it had to do something special for being in a docker container. Hoping it will not be necessary, but was asking ahead in case the loop device issue isn't the only challenge

If you want to continue making progress/further testing of sdm in docker, you could copy the IMG contents to a directory accessible in docker and then use the --directory switch. sdm doesn't use a loop device when mounting a directory.

simlu commented 11 months ago

Gotcha, all those make sense!

If you want to continue making progress/further testing of sdm in docker, you could copy the IMG contents to a directory accessible in docker and then use the --directory switch. sdm doesn't use a loop device when mounting a directory.

I was looking into that, but didn't fully understand how it works. How do I extract the img into a directory? Mainly that there are two partitions in the image, how do I deal with that? Could I just mount the main partition and point the "--directory" at the mounted partition?

gitbls commented 11 months ago

The code you provided doesn't work. It's close, but the 2nd mount gets an overlapping loop device error. Note that sdm requires both partitions to be mounted for the duration of the sdm customize.

Also, JFYI it creates an automatic loop device (at least for the first mount). If automatic loop devices don't work in docker this could be a non-starter approach.

w/ssdy/work# ISO_FULL_PATH=/ssd/work/2023-12-05-raspios-bookworm-arm64-lite.img ./x
Analysing ISO...
Mounting Boot Partition...
mkdir: cannot create directory β€˜/mnt/boot’: File exists
mount: /dev/loop0 mounted on /mnt/boot.
Mounting Main Partition...
mount: /mnt/main: overlapping loop device exists for /ssd/work/2023-12-05-raspios-bookworm-arm64-lite.img.
Unmounting...
umount: /mnt/main: not mounted.
simlu commented 11 months ago

Thank you for investigating! Great findings! It'll be a few day before I can look into this again, but based on what you are saying, there are a few things to look into

a) Can we get umount to work with non overlapping loop devices? b) Why does umount work in docker with an implicit loop device, but using loop devices explicitly does not.

I should have time to take another look after the weekend. Cheers, L~

gitbls commented 11 months ago

The problem is not umount. sdm needs to mount the root partition first to decide whether the boot partition should be mounted at /boot or /boot/firmware.

So, sdm mounts the root partition, and then it mounts the boot partition.

The 2nd mount (boot partition) fails with an overlap error.

sdm requires both partitions to be mounted for the entire customize process for various reasons. For instance,'apt upgrade' will fail for certain updates if the boot partition is not mounted.

gitbls commented 11 months ago

Gotcha, all those make sense!

If you want to continue making progress/further testing of sdm in docker, you could copy the IMG contents to a directory accessible in docker and then use the --directory switch. sdm doesn't use a loop device when mounting a directory.

I was looking into that, but didn't fully understand how it works. How do I extract the img into a directory? Mainly that there are two partitions in the image, how do I deal with that? Could I just mount the main partition and point the "--directory" at the mounted partition?

Here's a sketch of how to do it:

sudo sdm --mount /path/to/img.img
# Now in sdm --mount
rsync -a /mnt/sdm/ /path/to/newdir
exit # Exit the sdm --mount session

Then you can mount /path/to/newdir in your docker container somehow and use sdm --directory /path/to/img to operate on it.

This is the 2nd time you've mentioned "just the main partition". As I explained in another reply, sdm needs BOTH root and boot mounted. sdm itself doesn't have any requirements around the boot partition, except for the plugins that operate on it (bootconfig and quietness, for example), and the partition needs to be accessible during apt upgrade since any package could decide to update an existing file or provide a new file to the boot partition.

simlu commented 11 months ago

Apologies for the late reply. Hope you had a great Christmas time!

The code you provided doesn't work [...]

Here is the fixed code (the previous code was incorrect):

Expand ```sh ISO_FULL_PATH=... ROOT=/mnt/raspbian echo "Analysing ISO..." INFO=($(fdisk --bytes -lo Start,Size "${ISO_FULL_PATH}" | tail -n 2)) BOOT_START=$((${INFO[0]} * 512)) BOOT_SIZE=${INFO[1]} ROOT_START=$((${INFO[2]} * 512)) ROOT_SIZE=${INFO[3]} echo "Mounting Image..." sudo mkdir -p "${ROOT}" sudo mount -o loop,offset=${ROOT_START},sizelimit=${ROOT_SIZE} -t ext4 "${ISO_FULL_PATH}" "${ROOT}" sudo mount -o loop,offset=${BOOT_START},sizelimit=${BOOT_SIZE} -t vfat "${ISO_FULL_PATH}" "${ROOT}/boot" # DO WORK HERE # ... echo "Unmounting Image..." sudo umount -f "${ROOT}/boot/" sudo umount -f "${ROOT}" ```

Reference here

simlu commented 11 months ago

Alright, this is getting us somewhere. I've mounted the pi os image partitions as described above inside docker and then ran sudo cp -r "${ROOT}" "/tmp/root".

This resulted in the following

Expand ```sh > Command Line: /usr/local/bin/sdm --customize --plugin user:adduser=bls|password=mypassword --plugin L10n:host --plugin disables:piwiz --regen-ssh-host-keys --restart --directory /tmp/root * Host Information Hostname: 26657deefa68 uname: Linux 26657deefa68 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux os-release Name: Ubuntu Version: 20.04.4 (focal) Like OS: debian sdm Version: V10.2 * IMG Information Name: /tmp/root Date: 2023-12-11 RasPiOS Version: 12 RasPiOS Architecture: 64-bit aarch64 os-release Version: 12 % sdm will use systemd-nspawn on this 'x86-64' host Retry the command with --chroot if this fails sdm will run more slowly due to qemu emulation * Plugins included: * user Args: adduser=bls|password=mypassword * L10n Args: host * disables Args: piwiz > Directory '/tmp/root' has 43664416 1K-blocks (44.7GB, 41.6GiB) free > Copy sdm to /usr/local/sdm in the Directory * Start Phase 0 image customization > Run Plugins Phase '0' > Run Plugin 'user' (/mnt/sdm/usr/local/sdm/plugins/user) Phase 0 with arguments: 'adduser=bls|password=mypassword' * Plugin user: Start Phase 0 > Plugin user: Keys/values found: adduser: bls password: mypassword > Plugin user: Create user 'bls' home directory '/home/bls' so available for other plugins * Plugin user: Complete Phase 0 > Run Plugin 'L10n' (/mnt/sdm/usr/local/sdm/plugins/L10n) Phase 0 with arguments: 'host' * Plugin L10n: Start Phase 0 > Plugin L10n: Keys/values found: host: > Plugin L10n: Read L10n configuration from host system % /etc/default/keyboard missing; Keyboard set to 'us' > Plugin L10n: Load Localization (L10N) settings from running system > Plugin L10n: Keymap: us > Plugin L10n: Locale: en_US.UTF-8 > Plugin L10n: Timezone: Etc/UTC > Plugin L10n: WiFi Country: > Plugin L10n: Save L10n configuration ? Prefix L10n: Unrecognized keymap 'us' * Plugin L10n: Complete Phase 0 > Run Plugin 'disables' (/mnt/sdm/usr/local/sdm/plugins/disables) Phase 0 with arguments: 'piwiz' * Plugin disables: Start Phase 0 > Plugin disables: Keys/values found: piwiz: * Plugin disables: Complete Phase 0 * Phase 0 Completed * Enter image '/tmp/root' for Phase 1 Failed to open system bus: No such file or directory Attempted to remove disk file system under "/run/systemd/nspawn/propagate/sdm", and we can't allow that. umount: /mnt/sdm/boot (overlay) unmounted umount: /mnt/sdm (overlay) unmounted ```

When running sdm directly on the mounted partitions I get: 

sudo sdm --customize --plugin 'user:adduser=bls|password=mypassword' --plugin L10n:host --plugin disables:piwiz --regen-ssh-host-keys --restart --directory /mnt/raspbian
? Directory '/mnt/raspbian' is already mounted

Would be great if we could fix that, since copying everything over takes a lot of time and resources.

Let me know if you need anything else from my side. Cheers!

gitbls commented 11 months ago

Thanks! Looks promising. Will have an in-depth look into it.

simlu commented 11 months ago

I tried to play around with systemd-nspawn some more. In particular adding --register=no --keep-unit to prevent registration with system bus. But it really wasn't playing nice - issues with cgroup that I didn't fully understand and some other nonsense.

So here is the command from above, but with the --chroot option:

sudo sdm --batch --chroot --customize --plugin 'user:adduser=bls|password=mypassword' --plugin L10n:host --plugin disables:piwiz --regen-ssh-host-keys --restart --directory /tmp/root

Output

Expand ```sh * Start Configuration > Command Line: /usr/local/bin/sdm --batch --chroot --customize --plugin user:adduser=bls|password=mypassword --plugin L10n:host --plugin disables:piwiz --regen-ssh-host-keys --restart --directory /tmp/root * Host Information Hostname: b4714aceeea9 uname: Linux b4714aceeea9 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux os-release Name: Ubuntu Version: 20.04.4 (focal) Like OS: debian sdm Version: V10.2 * IMG Information Name: /tmp/root Date: 2023-12-11 RasPiOS Version: 12 RasPiOS Architecture: 64-bit aarch64 os-release Version: 12 % sdm will use chroot per --chroot on this 'x86-64' host sdm will run more slowly due to qemu emulation * Plugins included: * user Args: adduser=bls|password=mypassword * L10n Args: host * disables Args: piwiz > Directory '/tmp/root' has 48395116 1K-blocks (49.6GB, 46.2GiB) free > Copy sdm to /usr/local/sdm in the Directory * Start Phase 0 image customization > Run Plugins Phase '0' > Run Plugin 'user' (/mnt/sdm/usr/local/sdm/plugins/user) Phase 0 with arguments: 'adduser=bls|password=mypassword' * Plugin user: Start Phase 0 > Plugin user: Keys/values found: adduser: bls password: mypassword > Plugin user: Create user 'bls' home directory '/home/bls' so available for other plugins * Plugin user: Complete Phase 0 > Run Plugin 'L10n' (/mnt/sdm/usr/local/sdm/plugins/L10n) Phase 0 with arguments: 'host' * Plugin L10n: Start Phase 0 > Plugin L10n: Keys/values found: host: > Plugin L10n: Read L10n configuration from host system % /etc/default/keyboard missing; Keyboard set to 'us' > Plugin L10n: Load Localization (L10N) settings from running system > Plugin L10n: Keymap: us > Plugin L10n: Locale: en_US.UTF-8 > Plugin L10n: Timezone: Etc/UTC > Plugin L10n: WiFi Country: > Plugin L10n: Save L10n configuration ? Prefix L10n: Unrecognized keymap 'us' * Plugin L10n: Complete Phase 0 > Run Plugin 'disables' (/mnt/sdm/usr/local/sdm/plugins/disables) Phase 0 with arguments: 'piwiz' * Plugin disables: Start Phase 0 > Plugin disables: Keys/values found: piwiz: * Plugin disables: Complete Phase 0 * Phase 0 Completed * Enter image '/tmp/root' for Phase 1 /bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8) * Start Phase 1 image customization > Directory '/tmp/root' has 48388560 1K-blocks (49.5GB, 46.1GiB) free at start of Phase 1 image customization > Configure and enable sdm FirstBoot service (sdm-firstboot) > Start 'apt update' > Directory '/tmp/root' has 48388556 1K-blocks (49.5GB, 46.1GiB) free at start of 'apt update' > Directory '/tmp/root' has 48274888 1K-blocks (49.4GB, 46.0GiB) free at end of 'apt update' > Run Plugins Phase '1' > Run Plugin 'user' (/usr/local/sdm/plugins/user) Phase 1 with arguments: 'adduser=bls|password=mypassword' /bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8) * Plugin user: Start Phase 1 > Plugin user: Add user 'bls' > Plugin user: Add groups to user 'bls': dialout,cdrom,floppy,audio,video,plugdev,users,adm,sudo,users,input,netdev,spi,i2c,gpio > Plugin user: Enable sudo for user 'bls' > Plugin user: Copy skel files for user 'bls' > Plugin user: Set user 'bls' home directory '/home/bls' owner to 'bls:bls' > Plugin user: Set password for user 'bls' * Plugin user: Complete Phase 1 > Run Plugin 'L10n' (/usr/local/sdm/plugins/L10n) Phase 1 with arguments: 'host' /bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8) * Plugin L10n: Start Phase 1 % Plugin L10n: L10n item 'keymap' was not set > Plugin L10n: Set locale to 'en_US.UTF-8' Generating locales (this might take a while)... en_US.UTF-8... done Generation complete. > Plugin L10n: Set timezone to 'Etc/UTC' * Plugin L10n: Complete Phase 1 > Run Plugin 'disables' (/usr/local/sdm/plugins/disables) Phase 1 with arguments: 'piwiz' * Plugin disables: Start Phase 1 * Plugin disables: Complete Phase 1 * Phase 1 post-app installation/configuration > Start 'apt upgrade' > Directory '/tmp/root' has 48269892 1K-blocks (49.4GB, 46.0GiB) free at start of 'apt upgrade' > Directory '/tmp/root' has 48269888 1K-blocks (49.4GB, 46.0GiB) free at end of 'apt upgrade' > Start 'apt autoremove' > Directory '/tmp/root' has 48269888 1K-blocks (49.4GB, 46.0GiB) free at start of 'apt autoremove' > Directory '/tmp/root' has 48269888 1K-blocks (49.4GB, 46.0GiB) free at end of 'apt autoremove' > Directory '/tmp/root' has 48269888 1K-blocks (49.4GB, 46.0GiB) free at end of Phase 1 image customization * Phase 1 Completed > Run Plugins Phase 'post-install' > Run Plugin 'user' (/usr/local/sdm/plugins/user) Phase post-install with arguments: 'adduser=bls|password=mypassword' * Plugin user: Start Phase post-install * Plugin user: Complete Phase post-install > Run Plugin 'L10n' (/usr/local/sdm/plugins/L10n) Phase post-install with arguments: 'host' > Run Plugin 'disables' (/usr/local/sdm/plugins/disables) Phase post-install with arguments: 'piwiz' * Plugin disables: Start Phase post-install > Plugin disables: Disable running piwiz and userconfig on first boot per --disable piwiz * Plugin disables: Complete Phase post-install > Run graphics post-install configuration > Plugin sdm-phase1: No known Display Manager found > Plugin sdm-phase1: Set RasPiOS to B1:'console no login' for First Boot > Enable SSH > Plugin phase1: Enable SSH service via /boot/ssh > Directory '/tmp/root' has 48269880 1K-blocks (49.4GB, 46.0GiB) free at end of image customization * Customization complete * Batch Mode exit umount: /mnt/sdm/boot (overlay) unmounted umount: /mnt/sdm (overlay) unmounted ```

This already looks really good.

gitbls commented 11 months ago

A couple of questions:

  1. Confirming that this is only when mounting IMG files, and you don't need it for mounting, for instance, devices from within the docker container
  2. What's with this: /bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8) ?
simlu commented 11 months ago
  1. Yes, the only requirement for us is to be able to edit an image file.
  2. I am not sure. It might be because the docker container isn't set up correctly, but my fix attempts didn't work. I'll try some more and get back to you. This is a very harmless warning and shouldn't cause any issues.
gitbls commented 11 months ago

Thx.

Rather than me relying on you for testing this, you can save me some search-fu with a pointer to a simple docker setup that I can use for testing this. It seems to work well in my initial non-docker test, but haven't done much with it yet.

simlu commented 11 months ago

Just tested this, and if delete this line by running sudo sed -i '1184d' /usr/local/sdm/sdm-cparse, I don't need to copy the mount into a directory and can instead just work against the mount itself!

While certainly not ideal, this makes it somewhat usable for me now.

gitbls commented 11 months ago

Just tested this, and if delete this line by running sudo sed -i '1184d' /usr/local/sdm/sdm-cparse, I don't need to copy the mount into a directory and can instead just work against the mount itself!

While certainly not ideal, this makes it somewhat usable for me now.

Presumably this is to enable you to keep making progress, and not related to anything I'm doing?

For your planning purposes, I'm thinking later next week I should be able to release the next sdm update.

simlu commented 11 months ago

Presumably this is to enable you to keep making progress, and not related to anything I'm doing?

Sort of. Maybe there should be an option to allow "--directory" to be pointed to an already mounted device? I'm assuming the line in question exists as a check, however for me it prevents my (current) use case where I want to run sdm against a mounted image.

For your planning purposes, I'm thinking later next week I should be able to release the next sdm update.

Sounds good! I am unblocked for now, which is great.

gitbls commented 11 months ago

Yea, the check is there just to be careful. You've just proven that having it already mounted isn't problematic wrt doing a mount/bind on it.

I'm a bit uncomfortable removing it completely, as there could be unforseen issues. Thinking changing it to a %warning and continuing would be a better solution than full removal. Thoughts?

simlu commented 11 months ago

@gitbls I always like being explicit, so how about an additional flag that only takes mount points?

It looks like --mount is already used, so maybe --mount-point /path/to/mount/point?

Instead of running mount --bind $dimg $SDMPT you could just set them equal and run chroot / systemd-nspawn against the passed mount point directly?

It might also be time to refactor and add commands instead of flags on sdm, but I don't know enough about the syntax yet. E.g. sdm modify ..., sdm mount ..., sdm unmount etc. This would be a much bigger undertaking and breaking change though, so for another time

gitbls commented 11 months ago

While I understand your point about explicitness, adding --mount-point without getting rid of --directory doesn't seem like a win to me. It adds yet another thing to test/document/explain with questionable value from my perspective.

Someone else mentioned replacing switches with commands. I don't disagree with the sentiment, but there are other commands that do this. For instance: dpkg and update-alternatives.

There's nothing broken because of it and changing it would break literally everyone's sdm scripts. IOW it's a "How to piss users off and not create any new value for them" kind of change...Something that a certain company in Redmond has been known to do πŸ™„.

simlu commented 11 months ago

@gitbls Yeah, I don't disagree with any of that:

Changing switches to commands doesn't have to be a breaking change. You could slowly transition and deprecate the old way. But I would only do this if it is clearly better (i.e. you have distinct commands that don't join together somehow). Also it's really a question of time invested vs value gained. It was just a suggestion because some flags seemed to me like they are actually disjoined commands (and maybe they're not)

So I'm happy with just making it a warning until more people are running into it. You can revisit when the first person opens a ticket "why am I getting this warning?" =)

jacobsvante commented 10 months ago

@simlu Would you mind sharing your Dockerfile?

simlu commented 10 months ago

For sure, here are the files needed to run this locally

project/script.sh

# put any logic you want to run here
# ...

docker/Dockerfile

FROM cimg/node:18.0

RUN sudo apt update
# sdm install script
RUN curl -L https://raw.githubusercontent.com/gitbls/sdm/master/EZsdmInstaller | bash
# used by sdm for qemu emulation
RUN sudo apt install qemu-user-static binfmt-support
# used to resize image partitions
RUN sudo apt install udev

and

manage.sh

#!/bin/bash

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

echo "Starting docker container..."
docker build \
  --build-arg COMMAND=". script.sh" \
  -t sdm_build-container \
  --network="host" \
  $SCRIPT_DIR/docker/. &&
docker run \
  --rm \
  -v $SCRIPT_DIR/project:/user/project \
  -it --privileged sdm_build-container
simlu commented 10 months ago

Note that I couldn't get partition resizing to work with sdm, so I've did it custom for now.

Maybe we can revisit that once you release a new sdm version that uses the new mounting logic.

gitbls commented 10 months ago

Note that I couldn't get partition resizing to work with sdm, so I've did it custom for now.

Maybe we can revisit that once you release a new sdm version that uses the new mounting logic.

In order to get a few other things done, I postponed modifying the mounting logic for the following:

I think that's it. If the list is missing something, let me know after you've tested it*

These will get done, of course, just ran out of time to do them all at once.

simlu commented 10 months ago

No rush, I am no longer blocked on anything.

Happy to test and provide some demo code for documentation purpose once this is released.

gitbls commented 10 months ago

Just pushed V11. Please take it for a spin! Thx!

gitbls commented 10 months ago

In-depth look into the remaining issues WRT running in Docker. Here's my current findings:

The device contains 'ext4' signature and it may remain on the device. It is recommended to wipe the device with wipefs(8) or fdisk --wipe, in order to avoid possible collisions.


* `parted` plugin operating on an IMG: Same issue as burning to a file
* `parted` plugin operating on an SSD/SD Card: OK and should work in V11, pending your testing
* `sdm --shrink`: Same issue as burning to a file

If the three problematic scenarios are important to you, feel free to figure out how to make them work in Docker!
gitbls commented 9 months ago

Closing due to lack of activity. OP, please re-open if needed. Thx!

odi89 commented 6 months ago

@simlu @gitbls

Hey guys sorry for respawning a closed issue But i just want to ge this working on my mac in docker

I Have a question for you guys I have tried to make this work in docker on my m1 macbook pro. But i cant see to manage to do it, So i was wondering if it is possible when the host is a machine is a mac, or do you have to run debian/ubuntu as the host manchine.

This is what i have tried based on simlus successfull attempt, sorry if it is a bit long to read i want to go in detail

This is my host machine foler structure: [img]

drwxr-xr-x sebastian staff 192 B Sat May 11 21:40:14 2024 ο„• .
drwxr-xr-x sebastian staff 288 B Sat May 11 07:23:53 2024 ο„• ..
.rw-r--r-- sebastian staff 394 B Sat May 11 04:41:03 2024 ο’‰ develop.sh
drwxr-xr-x sebastian staff 256 B Sat May 11 21:38:26 2024 ο„• docker
.rwxr-xr-x sebastian staff 429 B Sat May 11 03:03:39 2024 ο’‰ manage.sh
drwxr-xr-x sebastian staff 160 B Sat May 11 04:30:37 2024 ο„• project

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) echo $SCRIPT_DIR echo "Starting docker container..."

Run for test

docker run -v $SCRIPT_DIR/docker:/user/project/docker -it --privileged sdm_build-container

docker run --rm -v /Users/seb/dev/hobby/auto-setup-docker/docker:/home/circleci/project -it --privileged sdm_build-container

* content of manage.sh (used tobild the image)
```bash
#!/bin/bash

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

echo "Starting docker container..."
# Run for test
docker build \
  --build-arg COMMAND=". script.sh" \
  --platform linux/arm64/v8 \
  -t sdm_build-container \
  --network="host" \
  $SCRIPT_DIR/docker/. &&
docker run \
  --rm \
  --storage-opt =10G \
  -v $SCRIPT_DIR/project:/user/project \
  -it --privileged sdm_build-container \

the first.img is a renamed --> 2023-12-11-raspios-bookworm-arm64-lite.img.xz
The .bashrc includes some simple scripts to try to work on the file
*bashrc
```bash
export SDM_BOOT_PATH="/home/circleci/project/sdm"
export ISO_FULL_PATH="/home/circleci/project/first.img"
export ROOT=/mnt/raspbian
export INFO=($(fdisk --bytes -lo Start,Size "${ISO_FULL_PATH}" | tail -n 2))
export BOOT_SIZE=${INFO[1]}
export ROOT_START=$((${INFO[2]} * 512))
export ROOT_SIZE=${INFO[3]}
export BOOT_START=$((${INFO[0]} * 512))
export BOOT_START_HARD=4194304
export BOOT_SIZE_HARD=268435456
export ROOT_START_HARD=272629760
export ROOT_SIZE_HARD=1832910848

This is what i have tried From inside the container on before hand

sudo sdm --mount /path/to/img.img # the raspberry pi image is in the container
# Now in sdm --mount
rsync -a /mnt/sdm/ /path/to/newdir 
exit # Exit the sdm --mount session
# I have then tried to copy the newdir folder out of the container and to the host machine

This is where the error commes I running sudo sh develoop.sh

Dockerfile  first.img  mount-script.sh  sdm
circleci@5b1a079029dc:~/project$ source .bashrc2
circleci@5b1a079029dc:~/project$ ls
Dockerfile  first.img  mount-script.sh  sdm
circleci@5b1a079029dc:~/project$ mount_image
Mounting Image...
circleci@5b1a079029dc:~/project$ ls
Dockerfile  first.img  mount-script.sh  sdm
circleci@5b1a079029dc:~/project$ run_sdm
% sdm will use qemu 'aarch64'
% Add and enable qemu binfmt for 'aarch64' processor architecture
% sdm will use chroot per --chroot on this 32-bit x86-64 host
* Enter Directory '/home/circleci/project/sdm'
/usr/local/sdm/sdm-cparse: line 1839: /usr/bin/stty: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1839: /usr/bin/stty: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1839: /usr/bin/stty: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1839: /usr/bin/stty: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1839: /usr/bin/stty: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1839: /usr/bin/stty: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1093: /usr/bin/mount: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1093: /usr/bin/mount: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1093: /usr/bin/mount: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1093: /usr/bin/mount: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1097: /usr/sbin/chroot: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 984: /usr/bin/grep: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 985: /usr/bin/umount: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1685: /usr/bin/findmnt: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1687: /usr/bin/findmnt: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1493: /usr/bin/findmnt: cannot execute binary file: Exec format errorhttps://github.com/gitbls
/usr/local/sdm/sdm-cparse: line 1493: /usr/bin/findmnt: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1706: /usr/bin/sync: cannot execute binary file: Exec format error

The run_sdm is a bash script in the .bashrc file

run_sdm() {

    if [ $# -lt 1 ]; then
        directory=$TEMP_ROOT  
    else
        directory="$1"     # Use the passed argument
    fi
sudo sdm --batch --chroot --customize --plugin 'user:adduser=bls|password=mypassword' --plugin L10n:host --plugin disables:piwiz --regen-ssh-host-keys --restart --directory $directory

when i run this command i get much errors in the 

/usr/local/bin/sdm: line 825: /usr/bin/cp: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 157: /usr/bin/chown: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 158: /usr/bin/chmod: cannot execute binary file: Exec format error
/usr/local/bin/sdm: line 828: /usr/bin/chown: cannot execute binary file: Exec format error
* Start Configuration
/usr/local/sdm/sdm-cparse: line 397: /usr/bin/file: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 50: /usr/bin/fold: cannot execute binary file: Exec format error

* Host Information
/usr/local/bin/sdm: line 840: /usr/bin/hostname: cannot execute binary file: Exec format error
   Hostname:
/usr/local/bin/sdm: line 841: /usr/bin/uname: cannot execute binary file: Exec format error
   uname:
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
   os-release Name:
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
   Version:
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
   sdm Version: V12.0
* IMG Information
   Name: /home/circleci/project/sdm
/usr/local/sdm/sdm-cparse: line 376: /usr/bin/grep: cannot execute binary file: Exec format error
   Date:
   RasPiOS Version:
   RasPiOS Architecture: 32-bit armhf
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
   os-release Version:

First of all i am so sorry that this post is all over the place, but i wanted to ensure you that i have tried much(and more than is described here go get this to work) but do you guys now if it is even possible to run sdm in a docker container if the host machine is a mac

have tried to mount the image then Rsynced the directory the director to the host machine, and then mount that folder onto the container. But when i try to run any commands the docker breaks and

gitbls commented 6 months ago

@odi89 I don't have a Mac, and I don't use docker. If you're willing to try various things I suggest, then we can probably figure out what's going on.

One immediate question:. You said it's an M1 Mac. You also said that first.img is a copy of an arm64 RasPiOS. Yet, in the log output I see:

   Date:
   RasPiOS Version:
   RasPiOS Architecture: 32-bit armhf

Please clarify what you're actually using.

odi89 commented 6 months ago

@gitbls Thank you for your fast response and helpfull demenaour i will happily try any thing you suggest. And further down in the process i will try to document my comments so it is easier for you to help and maybe some others who tries to get sdm to run inside docker will find it helpfull/usefull further down the line.

I saw your last comment. You are asbolutely right, first.img was wrong image. I now downloaded 2024-03-15-raspios-bookworm-arm64-lite.img and tried again

Okay this is the process i took

Started the docker container

mounted the ISO following your steps

sudo sdm --mount 2024-03-15-raspios-bookworm-arm64-lite.img
rsync -a /mnt/sdm/ /tmp/root 
exit # Exit the sdm --mount session

Then from outside the container i synced the root directory back to my host machine(mac ) so i dont have to do this every time

# From host machine
##/Users/sebastian/dev/hobby/auto-setup-docker/docker
sudo docker cp  082dabf64821:/tmp/root ./sdm # This copies the mounted isofile from inside the container --> to the hostmachine in a folder called /sdm

Back inside the container i tried to run the newly mountes iso image from /tmp/root but stil got this

run_sdm_from_directory # Bash script running ls sudo sdm --batch --chroot --customize --plugin 'user:adduser=bls|password=mypassword' --plugin L10n:host --plugin disables:piwiz --regen-ssh-host-keys --restart --directory $directory

/usr/local/sdm/sdm-cparse: line 50: /usr/bin/fold: cannot execute binary file: Exec format error

* Host Information
/usr/local/bin/sdm: line 840: /usr/bin/hostname: cannot execute binary file: Exec format error
   Hostname:
/usr/local/bin/sdm: line 841: /usr/bin/uname: cannot execute binary file: Exec format error
   uname:
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
   os-release Name:
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
   Version:
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
   sdm Version: V12.0
* IMG Information
   Name: /tmp/root
/usr/local/sdm/sdm-cparse: line 376: /usr/bin/grep: cannot execute binary file: Exec format error
   Date:
   RasPiOS Version:
   RasPiOS Architecture: 32-bit armhf
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
   os-release Version:
% sdm will use qemu 'aarch64'
% Add and enable qemu binfmt for 'aarch64' processor architecture
% sdm will use chroot per --chroot on this 32-bit x86-64 host
* Plugins selected:
   * user
       Args: adduser=bls|password=mypassword
   * L10n
       Args: host
   * disables
       Args: piwiz
/

I will no experiment with other images tried to download the This 2024-03-15-raspios-bookworm-arm64-full.img.xz version to test with that Is it something that i am missing? This is the content of (maybe) some relevant file

manage.sh

#!/bin/bash

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

echo "Starting docker container..."
# Run for test
docker build \
  --build-arg COMMAND=". script.sh" \
  --platform linux/arm64/v8 \
  -t sdm_build-container \
  --network="host" \
  $SCRIPT_DIR/docker/. &&
docker run \
  --rm \
  --storage-opt =10G \
  -v $SCRIPT_DIR/project:/user/project \
  -it --privileged sdm_build-container \

develop.sh
```bash
#!/bin/bash

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
echo $SCRIPT_DIR
echo "Starting docker container..."
# Run for test
#docker run -v $SCRIPT_DIR/docker:/user/project/docker -it --privileged sdm_build-container
docker run --rm -v /Users/sebastian/dev/hobby/auto-setup-docker/docker:/home/circleci/project -it --privileged sdm_build-container

Dockerfile

FROM cimg/node:18.0
# Install dependencies
RUN sudo apt-get update && sudo apt-get install -y \
    curl \
    qemu-user-static \
    binfmt-support \
    udev
# Install sdm
RUN curl -L https://raw.githubusercontent.com/gitbls/sdm/master/EZsdmInstaller | bash
gitbls commented 6 months ago

What is the result if you try to use sdm on the directory tree outside of docker (but still on the Mac)? IOW just remove docker completely from the equation.

odi89 commented 6 months ago 
❯ sudo sdm --batch --chroot --customize --plugin 'user:adduser=bls|password=mypassword' --plugin L10n:host --plugin disables:piwiz --regen-ssh-host-keys --restart --directory ./sdm
Password:
/usr/local/bin/sdm: line 311: --plugin: command not found
/usr/local/bin/sdm: line 315: ${1,,}: bad substitution
/usr/local/bin/sdm: line 365: ${dimg,,}: bad substitution
? No command specified (--aptmaint --burn, --customize, --explore, --mount, --runonly)

another command

❯ sdm --mount 2024-03-15-raspios-bookworm-arm64-full.img
/usr/local/bin/sdm: line 315: ${1,,}: bad substitution
/usr/local/bin/sdm: line 365: ${dimg,,}: bad substitution
? No command specified (--aptmaint --burn, --customize, --explore, --mount, --runonly)

Edit2: This is why i started the process of getting to doing this in docker in the first place. I could not get it sdm to work on my mac

gitbls commented 6 months ago

As I mentioned, I don't have a Mac, nor am I looking over your shoulder to see what you might be doing wrong, so you're going to have to resolve this issue yourself. To me, the errors you showed above look like the main sdm file (sdm) is corrupted somehow.

Another approach to using sdm on a Mac is to use a Debian VM, as ljames8 did here: https://github.com/gitbls/sdm/issues/175

odi89 commented 6 months ago

Thank you for your time<3

gitbls commented 6 months ago

@odi89

Thank you for your time<3 Sorry...you're using two technologies that I don't, most importantly, the Mac.

If you're interested in trying to chase down why sdm doesn't run on the Mac, I'm willing to help, but it will take some investigation and likely a fair amount of work/testing on your part. The failure you showed with the two sdm commands running on Bash on the Mac (sans docker) indicate something strange and fundamental is wrong.

One last question: What version of bash is on your mac? bash --version

odi89 commented 6 months ago

@gitbls sorry. for late respons, did not see this. I solved most of my use cases, but i did not get to use sdm.

But i would love to help out, and i would love to get sdm working in docker on mac or just nativly on mac.

you are right, Mac now uses zsh as default shell.

i tried to use bash instead, but same problem

auto-setup-docker/docker on ξ‚  main [!] on ☁️  sebastian.melbye@drdropin.no
❯ sdm mount 2024-03-15-raspios-bookworm-arm64-full.img
? Please run as root: sudo /usr/local/bin/sdm mount 2024-03-15-raspios-bookworm-arm64-full.img

auto-setup-docker/docker on ξ‚  main [!] on ☁️  sebastian.melbye@drdropin.no
❯ sudo sdm 2024-03-15-raspios-bookworm-arm64-full.img
/usr/local/bin/sdm: line 315: ${1,,}: bad substitution
/usr/local/bin/sdm: line 365: ${dimg,,}: bad substitution
? No command specified (--aptmaint --burn, --customize, --explore, --mount, --runonly)

auto-setup-docker/docker on ξ‚  main [!] on ☁️  sebastian.melbye@drdropin.no
❯ bash
bash-5.2$ sudo sdm mount 2024-03-15-raspios-bookworm-arm64-
/usr/local/bin/sdm: line 315: ${1,,}: bad substitution
/usr/local/bin/sdm: line 365: ${dimg,,}: bad substitution
? No command specified (--aptmaint --burn, --customize, --explore, --mount, --runonly)
bash-5.2$ sudo sdm mount 2024-03-15-raspios-bookworm-arm64-full.img
/usr/local/bin/sdm: line 315: ${1,,}: bad substitution
/usr/local/bin/sdm: line 365: ${dimg,,}: bad substitution
? No command specified (--aptmaint --burn, --customize, --explore, --mount, --runonly)
bash-5.2$

bash-5.2$ sudo sdm --mount 2024-03-15-raspios-bookworm-arm64-full.img
/usr/local/bin/sdm: line 315: ${1,,}: bad substitution
/usr/local/bin/sdm: line 365: ${dimg,,}: bad substitution
? No command specified (--aptmaint --burn, --customize, --explore, --mount, --runonly)
bash-5.2$

again, sorry for late reply! Will check this thread for further information=)

Cheers, from your Mac guy in Norway

gitbls commented 5 months ago

@odi89 Would you please try the following commands on the console and post the output here?

# start a bash shell from zsh
which bash
/bin/bash
# Print the bash version
bash-5.2$ bash --version
bash-5.2$ getopt --version
bash-5.2$ sudo sdm --version
# Now let's see if the ${var,,} construct works
bash-5.2$ s="ABCDEFG"
bash 5.2$ echo ${s,,}

Thx!

6/27/2024: @odi89 Still waiting on your input to the above. Thx!

odi89 commented 3 months ago

@gitbls So sorry for taking so long. And thanks for keeping up. I did manage to get this to work to my liking Running and customizing images but not burning the image to disk.

Using a docker container with debian and theese dependencies (described in another thread)

    sudo \
    curl \
    unzip \
    fdisk \
    dosfstools \
    rsync \
    parted \
    kpartx \
    qemu-utils \
    qemu-user-static \
    qemu-system-arm \
    qemu-efi-aarch64 \
    ipxe-qemu \
    qemu-efi-arm \
    qemu-system-gui \
    systemd-container \
    binfmt-support \
    util-linux \
    xz-utils \
    zip \
    bzip2 \
    file \
    less \
    && rm -rf /var/lib/apt/lists/*
gitbls commented 3 months ago

I'm running into the same issue, I tried installing Bash 5.2 via Homebrew and still getting the same error

Sorry about that. If you're on a Mac, I have no explanation other than there's an issue with the bash you're using, since sdm works perfectly fine on RasPiOS, Windows WSL, X64-based Ubuntu and Debian.

To me, the fact that this error only shows up on Macs is highly indicative of a problem with the Mac's Bash.

I've asked previously in this thread for some information that would be helpful for me to identify the problem, but so far nobody has provided it.

If you're interested and willing to work with me to chase this down, then please start by providing the information I requested here.

aendra-rininsland commented 1 month ago

@gitbls Sorry for the huge delay getting back to you with this!

bash-3.2$ which bash

/bin/bash

bash-3.2$ bash --version

GNU bash, version 3.2.57(1)-release (arm64-apple-darwin23)
Copyright (C) 2007 Free Software Foundation, Inc.

bash-3.2$ getopt --version

--

bash-3.2$ sudo sdm --version

/usr/local/bin/sdm: line 315: ${1,,}: bad substitution
/usr/local/bin/sdm: line 365: ${dimg,,}: bad substitution
? No command specified (--aptmaint --burn, --customize, --explore, --mount, --runonly)

bash-3.2$ s="ABCDEFG" bash-3.2$ echo ${s,,}

bash: ${s,,}: bad substitution

If I switch to the Homebrew version...

% which bash

/opt/homebrew/bin/bash

% bash bash-5.2$ bash --version


GNU bash, version 5.2.32(1)-release (aarch64-apple-darwin23.4.0)
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.


> bash-5.2$ getopt --version

--


> bash-5.2$ sudo sdm --version

/usr/local/bin/sdm: line 315: ${1,,}: bad substitution /usr/local/bin/sdm: line 365: ${dimg,,}: bad substitution ? No command specified (--aptmaint --burn, --customize, --explore, --mount, --runonly)


> bash-5.2$ s="ABCDEFG"
> bash-5.2$ echo ${s,,}

abcdefg


...I also tried installing getopt from Homebrew:
> % brew install gnu-getopt
> % echo 'export PATH="/opt/homebrew/opt/gnu-getopt/bin:$PATH"' >> ~/.bashrc
> % echo 'export PATH="/opt/homebrew/opt/gnu-getopt/bin:$PATH"' >> ~/.zshrc

...Closed all shells...

> bash-5.2$ getopt --version

getopt from util-linux 2.40.2%



...But same error when I run `sudo sdm --version`.
gitbls commented 1 month ago

@gitbls Sorry for the huge delay getting back to you with this!

bash-3.2$ which bash

/bin/bash

bash-3.2$ bash --version

GNU bash, version 3.2.57(1)-release (arm64-apple-darwin23)
Copyright (C) 2007 Free Software Foundation, Inc.

bash-3.2$ getopt --version

 --

bash-3.2$ sudo sdm --version

/usr/local/bin/sdm: line 315: ${1,,}: bad substitution
/usr/local/bin/sdm: line 365: ${dimg,,}: bad substitution
? No command specified (--aptmaint --burn, --customize, --explore, --mount, --runonly)

bash-3.2$ s="ABCDEFG" bash-3.2$ echo ${s,,}

bash: ${s,,}: bad substitution

If I switch to the Homebrew version...

% which bash

/opt/homebrew/bin/bash

% bash bash-5.2$ bash --version

GNU bash, version 5.2.32(1)-release (aarch64-apple-darwin23.4.0)
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

bash-5.2$ getopt --version

--

bash-5.2$ sudo sdm --version

/usr/local/bin/sdm: line 315: ${1,,}: bad substitution
/usr/local/bin/sdm: line 365: ${dimg,,}: bad substitution
? No command specified (--aptmaint --burn, --customize, --explore, --mount, --runonly)

bash-5.2$ s="ABCDEFG" bash-5.2$ echo ${s,,}

abcdefg

...I also tried installing getopt from Homebrew:

% brew install gnu-getopt % echo 'export PATH="/opt/homebrew/opt/gnu-getopt/bin:$PATH"' >> ~/.bashrc % echo 'export PATH="/opt/homebrew/opt/gnu-getopt/bin:$PATH"' >> ~/.zshrc

...Closed all shells...

bash-5.2$ getopt --version

getopt from util-linux 2.40.2%

...But same error when I run sudo sdm --version.

Simply put: I don't understand what point you're trying to make.

I see that the default bash is super-old ( GNU bash, version 3.2.57(1)-release (arm64-apple-darwin23)), while Debian Bookworm has 5.2.15(1), so that's clearly a problem.

I don't understand what "getopt" is, or has anything to do with any of this.

The issue with sdm on MacOS is only going to get fixed if someone wants to step up and work closely with me on this, committing to test things that I ask and provide timely responses.

aendra-rininsland commented 1 month ago

@gitbls Apologies, I was simply trying to provide as detailed an error case as possible given that's what you asked for here.

I tried it with both the system Bash as well as one installable from Homebrew because the version of Bash that ships with MacOS is brutally old and a lot of stuff attached to with it is incompatible with what you'd expect in the average Linux distribution; you can install a more recent Bash from Homebrew (5.2.32 as per the output), but it will generally lack a lot of the niceties you'd expect (ergo, installing gnu-getopt, which is ultimately the bash completion extensions provided by getopt, I think?). I only investigated it because you specifically asked for that in that snippet you requested people run to troubleshoot.

I'm happy to troubleshoot it with you but I've sort of moved on to running sdm exclusively in Linux, where it works pretty flawlessly; mainly providing the output there because I remembered this issue and thought it might be helpful because nobody else seemed to be as per your request.

gitbls commented 1 month ago

@gitbls Apologies, I was simply trying to provide as detailed an error case as possible given that's what you asked for here.

I tried it with both the system Bash as well as one installable from Homebrew because the version of Bash that ships with MacOS is brutally old and a lot of stuff attached to with it is incompatible with what you'd expect in the average Linux distribution; you can install a more recent Bash from Homebrew (5.2.32 as per the output), but it will generally lack a lot of the niceties you'd expect (ergo, installing gnu-getopt, which is ultimately the bash completion extensions provided by getopt, I think?). I only investigated it because you specifically asked for that in that snippet you requested people run to troubleshoot.

I'm happy to troubleshoot it with you but I've sort of moved on to running sdm exclusively in Linux, where it works pretty flawlessly; mainly providing the output there because I remembered this issue and thought it might be helpful because nobody else seemed to be as per your request.

Apologies if I misunderstood your intent. As you might infer, I'm frustrated by this issue.

Since the ${symbol,,} construct works in isolation from the command line but fails in sdm, I'll need to do a bit more code review, and hope to find a few simple tests that I will ask you to try.

Could you confirm the sdm version that you were using?

Thanks!

aendra-rininsland commented 1 month ago

No worries, totally understood! I can get that it's especially annoying given how well sdm runs literally everywhere else and this is an issue very much due to Apple's lacklustre UNIX implementation!

According to my reading of the /usr/local/bin/sdm's source (I can't run sudo sdm --version because it crashes due to the aforementioned issue; looking at line 196), I think I'm on 12.6.