File "alez-downloader.sh" referenced in ISO self-creation instructions txt no longer exists in repo

[pmarreck]

https://github.com/danboid/ALEZ/blob/cbda8a4d42017c32cf66cbff4cea665d73de586d/create-alez-iso.txt#L48

Basically, I want to play with a build of this. ;)

[pmarreck]

I did investigate the git history and I see this closed issue referenced https://github.com/danboid/ALEZ/issues/42 which seems to indicate that Docker is now the recommended way to get an updated ISO file?

[pmarreck]

Update: I tried the Docker method and couldn't get a build to complete, error on step 14/17:

~/ALEZ> docker build -t alezbuild .
# ...
Step 14/17 : RUN sed -i '\;cp\ \"\${work_dir}/x86_64/airootfs/usr/share/licenses/amd-ucode/LICENSE\";d' "${ALEZ_BUILD_DIR}/iso/build.sh" &&     sed -i '\;\"\${work_dir}/iso/${install_dir}/boot/amd_ucode.LICENSE\";d' "${ALEZ_BUILD_DIR}/iso/build.sh"
 ---> Running in 1fc2a727abd0
sed: can't read /opt/alez/iso/build.sh: No such file or directory

[pmarreck]

I'd be willing to help update this and submit a PR but I'm more of an Elixir dev and not a sysadmin (for example, I didn't understand SOME of the bash syntax in alez.sh, but I could probably figure it out, just a lot quicker with someone else... also, so much of that could have been done more readably (think: long-term maintenance) in another scripting language, but I get (kinda) why people still write Bash scripts... also, docker has been updated quite a bit since this so for example the VOLUME directive in the Dockerfile is very much actively discouraged, etc..). Pinging @johnramsden , not sure if you're still motivated and you're probably busy with school but let me know if you want to pow-wow since you made all the last commits to this.

Lastly, I'd recommend going with ONE OFFICIAL way of generating the ISO yourself instead of "maybe this bash script or maybe this dockerfile"

Lastly lastly, did something political happen with regards to ZFS and Linux distro support a year ago (other than the usual "the ZFS license is nonpermissive/still owned by Oracle")? If so, I missed the memo. A lot of zfs-related Linux projects I'm finding on ye olde Internets seem to have not been updated in over a year... might have something to do with openzfs 2.0 complicating things, or pandemic, or loss of interest, or who knows what...

[johnramsden]

Sorry for the late reply on this. I don't really have a lot of time to continue maintaining this project, and I don't believe @danboid does either. That said I'm perfectly happy to assist someone who would like to fix up whatever remaining issues there are. As well as review any new code that is required.

It was working fine before, but due to some changes to archiso, the way we were generating ISOs stopped working and I didn't have the motivation or time to get it working again.

I can try to reproduce your error this weekend, and see what might be the problem in building the image.

[pmarreck]

I get that... appreciate the response regardless! And I'm paying attention! Let me know if I can be of any assistance

[johnramsden]

@pmarreck OK, it now builds, archiso switched to mkarchiso, so changes were needed. I haven't tested building an ISO, but we're closer now, the image builds. Its possible more mkarchiso related changes are needed.

[pmarreck]

Nice, I will play with it in the next couple of days!

[pmarreck]

I was able to build it via Docker!

Will try building an ISO next but what you saw might happen is likely

[pmarreck]

[pmarreck@archvm ALEZ]$ docker run -it --rm --privileged -v /home/pmarreck/Documents/ALEZ/iso_out:/opt/alez/iso/out alezbuild
[mkarchiso] ERROR: No profile specified
usage: mkarchiso [options] <profile_dir>

seems easy enough to fix, just specify a profile_dir, but I have no idea what the concept of a "profile" is regarding mkarchiso ;)

[pmarreck]

https://www.reddit.com/r/archlinux/comments/jkg40r/archiso_build_error/

This might have the solution on how to invoke it properly now; this person had to find the answer in the mailing list archives, so I guess it wasn't terribly well documented and/or clear. Unfortunately I'm currently out of time

[pmarreck]

I got it to work!! I added a file in the ALEZ directory called "profiledef.sh" which is basically just an unmodified copy of the releng profile (the default non-minimized ISO definition... You may want to modify values to conform to the project name etc.):

#!/usr/bin/env bash
# shellcheck disable=SC2034

iso_name="archlinux"
iso_label="ARCH_$(date +%Y%m)"
iso_publisher="Arch Linux <https://www.archlinux.org>"
iso_application="Arch Linux Live/Rescue CD"
iso_version="$(date +%Y.%m.%d)"
install_dir="arch"
bootmodes=()

I then just modified the last CMD line in the Dockerfile to pass the working directory in as a parameter:

CMD ["/usr/bin/mkarchiso", "./"]

Then I rebuilt the Docker image and reran it with:

docker build -t alezbuild .

docker run -it --rm --privileged -v /home/pmarreck/Documents/ALEZ/iso_out:/opt/alez/iso/out alezbuild

It took a few minutes (and seemed to do nothing for the first 2 minutes or so, FYI) but then the fans spun up and soon enough I had an ISO outfile!

[johnramsden]

If everything works a PR would be much appreciated

[pmarreck]

I booted off the image and trying to run the “alez” command at the root prompt gives a zsh permission error… as root… which I’ve never encountered before.

Not sure how to fix that but I can send over a WIP PR soon

[pmarreck]

PR: https://github.com/danboid/ALEZ/pull/70

Also, in previous comment I said "zfs" when I meant "zsh" (now edited). Important difference, that :)

[pmarreck]

I'd love to come up with an automated test that somehow persists through reboots (probably just by driving this from outside via Docker) but I'm not sure how to do procedural I/O with an interactive script running inside a Docker image... would be cool, though... it's probably googleable, maybe something like this to convert the generated ISO into a Docker image and then drive that from the outside somehow: https://www.techrepublic.com/article/how-to-convert-an-iso-to-a-docker-image/

[pmarreck]

I added test code in a new commit that builds the docker image, runs it which generates the ISO, and then runs the ISO using qemu and a helper script (new deps are listed inside the test script). At the end of this I was able to immediately duplicate the issue (just run alez once qemu fires up the ISO). This should make it trivial for you to look at it!

[pmarreck]

Progress update: I got past the security issue and got the installer to run once the image was built and up by running bash /usr/local/bin/alez; apparently zsh (which seems to be the default now) won't permit a shebang line (in this case bash) that specifies a different shell in a script you run directly... which I guess makes sense, security-wise... maybe it's a shell option I don't know about that drops this new security requirement? Oh, and I also had to install dialog using pacman, because that's apparently not part of the built ISO now (and it thus errored on it).

THEN it ran!

I actually don't know offhand how to make those changes (add dialog to the binaries on the built ISO, and convince zsh that it's OK to directly run a bash script that shebang-specifies bash), but this may make it even easier for you at least

[pmarreck]

Note: The keyserver URL's in the script need to be updated since the sks keyservers are apparently a GDPR casualty and have been taken down. keyserver.ubuntu.com may work; see https://www.reddit.com/r/archlinux/comments/oybjij/pgp_keys_fubar/

But this may not work as I'm currently running the script in a "debug mode" having removed all redirects of output to /dev/null, and it can't find many (all?) of the signatories via the ubuntu keyserver and it seems to take FOREVER to update/verify (there are hundreds (?), and each takes a few seconds to verify... This seems a little unreasonable, of course)

[johnramsden]

Thank you for your hard work, i've just been really busy these past few weeks. I hope to take a look at your PR when I get a chance.

[pmarreck]

Don't worry! Just pushed up an additional commit that should aid when debugging the script. Basically if you run alez with DEBUG=true alez or TEST=true alez then some additional status output should be generated (could also redirect it to a logfile, etc.) and command output that was normally discarded to /dev/null actually gets output to STDOUT (maybe STDERR would be better, or a logfile, but it's easily changeable globally now)

[pmarreck]

Have to say I'm not used to coding without tests (for years now) and it's quite easy to accidentally introduce bugs when refactoring anything without a test suite (especially in Bash!); I investigated ways to test bash shell scripts (BATS was one thing) but the rabbit hole seemed a little deeper of a commitment than I felt comfortable with changing/adding

[johnramsden]

I emailed you btw.

[pmarreck]

Ha, I've been terrible at email management. I feel like I added some problems potentially in my most recent commit (Bash, arggghhhh) after testing, so I'm going to bang on it today some more. I really need to get something with test assertions working though. I don't even know why I'm obsessed with making this work lol. (Could just use Ubuntu's zfs support... but it's Ubuntu) I will check my email EDIT: Answered email!

[pmarreck]

Pushed up another commit with some changes/fixes and cleanups. Added the option to specify a virtual block device to the qemu ISO runner.

The ./test script (which is not a suite, just something to get the whole thing running without needing to try it on a real thumbdrive on real hardware) works and everything builds and runs in Docker/QEMU, but the alez script fails at the partition selection (possibly because no partition ending in -part[1/2] was listed, maybe due to idiosyncrasy with specifying virtual block devices in qemu, currently investigating).

Still investigating what an actual end to end test suite might look like, in particular 1) what test library to use; ShellSpec looks the most full-featured but I don't like its Cucumber-like wordiness (Cucumber once left a bad taste in my mouth years ago), 2) how to drive the dialog command from a unit test and assert on its output, 3) how to send IO to/from a running QEMU VM and assert on things in a test script (I saw some capability here but need to investigate further)

Fun weekend times. ;) I know you're busy, no rush, look forward to your thoughts when you have time.