Refine the utils scripts a bit.

[arlaneenalra]

While attempting to build out a dev branch of zeptoforth on a macOs machine (Ventura 13.5 on an M1 to be specific) I ran into a few incompatibilities with the that tripped me up for a while. In particular:

• The shipped version of screen mishandles \n and sends it as the characters \ and n instead of a newline. (Using Homebrew's packaged version appears to fix this. The code below adds a test that checks for the errant version, warns users that it won't work, and recommends installing the Homebrew package instead. The check should only react on macOs
• The scripts could partial fail and continue executing. I added set -e which should cause the scripts to crash out for most failures instead of silently continuing in a broken state.
• pySerial needed to be installed globally or users needed to create a virtual environment for python by hand, but there were no requirements files etc. I added a requirements.txt file and shell wrapper to codeloader3.py and make_uf2.py that manges a virtual environment mostly automatically for those scripts. There may be others that could use a similar treatment, but those were the ones I saw.
• Added a common.sh script that handles the screen session handling with shell functions to drive the ihex and ihex.minidict downloads. This gets rid of a bunch of duplication. This also has a trap to clean up the screeenlog.0 file at the end and shutdown any running screen sessions. This could be made nicer by having it name the session being used, but I'm effectively out of time to do that.
• Adds a note about creating USB console images on the Raspbery Pi Pico. I kept running into issues with the console crashing out around the ihex step so I added a heuristic hook to that script that recommends rebooting the chip and using download_uf2_image.sh to finish those images.
• Miscellaneous code clean up bits and updates to the README.md

In theory, I didn't change the calling semantics of the scripts (though they should be less sensitive to their working directory now) and I only have a Raspberry Pi Pico to play with so I can't test the other chips. So far I've built the full and full_usb images using these.

[tabemann]

Chris,

Thanks for doing all this work ─ I don't have access to a Mac environment so I would not have been able to test this. However, I have a small quibble ─ could you modify your scripts to use plain Bourne shell and not bash, because xBSD's often do not come with bash out of the box?

Thanks,

Travis

[arlaneenalra]

I can give it a shot. I'm not sure if I have a system that actually has a real bourne shell on it (most Linux systems I've worked with symlink sh to something Bourne like ). Is there a particular xBSD you use for testing?

On Tue, Aug 22, 2023, 01:01 tabemann @.***> wrote:

Chris,

Thanks for doing all this work ─ I don't have access to a Mac environment so I would not have been able to test this. However, I have a small quibble ─ could you modify your scripts to use plain Bourne shell and not bash, because xBSD's often do not come with bash out of the box?

Thanks,

Travis

— Reply to this email directly, view it on GitHub https://github.com/tabemann/zeptoforth/pull/60#issuecomment-1687492094, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACYAPYCQEP4I67W4CRLU6LXWRDNLANCNFSM6AAAAAA3XUXXMQ . You are receiving this because you authored the thread.Message ID: @.***>

[tabemann]

I can give it a shot. I'm not sure if I have a system that actually has a real bourne shell on it (most Linux systems I've worked with symlink sh to something Bourne like ). Is there a particular xBSD you use for testing?

I do not use xBSD myself, but I know people who do use xBSD (primarily FreeBSD). To use what is almost Bourne shell rather than Bash just specify #!/bin/sh at the top of your script. (Bash detects this and enters a fallback emulation mode when it is specified.)

[arlaneenalra]

True enough, though I just built a freebsd vm to play with. If I can't make that work, I'll use the bash emulation as you suggest.

On Tue, Aug 22, 2023, 18:54 tabemann @.***> wrote:

I can give it a shot. I'm not sure if I have a system that actually has a real bourne shell on it (most Linux systems I've worked with symlink sh to something Bourne like ). Is there a particular xBSD you use for testing?

I do not use xBSD myself, but I know people who do use xBSD (primarily FreeBSD). To use what is almost Bourne shell rather than Bash just specify #!/bin/sh at the top of your script. (Bash detects this and enters a fallback emulation mode when it is specified.)

— Reply to this email directly, view it on GitHub https://github.com/tabemann/zeptoforth/pull/60#issuecomment-1689068894, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACYAP44OALCX24FIZYECN3XWVBDLANCNFSM6AAAAAA3XUXXMQ . You are receiving this because you authored the thread.Message ID: @.***>

[arlaneenalra]

Hmm I think I have some variable scoping issues that didn't show up in bash .. trying to get those sorted out ...

[arlaneenalra]

Ok, how's that? I can't test the stuff with st-flash calls and my FreeBSD vm wound up being unstable in proxmox. ( Still need to figure that out but it's unrelated to this). I noticed two things with it though:

• screen was injecting timestamps into the log file ( added code to disable that)
• make misbehaved without the .OBJDIR line.

Past that, I had to switch back to my Mac for testing.

[tabemann]

I have tried it out with my Pico W and it works for me. The only thing that I noticed was that I had to install the python3.11-venv Debian package, which I had not expected. However, using that does improve things w.r..t. avoidng having users manually install pySerial. (On the other hand, some people are a bit paranoid about PIP, seeing that as a security thread for some reason.)

[arlaneenalra]

I'm not sure of a good answer for that ... is pySerial normally installed via a debian package or manually? I'm actually surprised venv is a separate package, I thought that was a core part of python now.

Either way, adding a fallback or flag/env variable to make the venv stuff optional wouldn't be too difficult. The other nice thing about venv's is that if something ever goes wrong with it, the right answer is almost always delete the venv and rebuild it. We use a technique like this all the time at the day job.

On Wed, Aug 23, 2023, 14:35 tabemann @.***> wrote:

I have tried it out with my Pico W and it works for me. The only thing that I noticed was that I had to install the python3.11-venv Debian package, which I had not expected. However, using that does improve things w.r..t. avoidng having users manually install pySerial. (On the other hand, some people are a bit paranoid about PIP, seeing that as a security thread for some reason.)

— Reply to this email directly, view it on GitHub https://github.com/tabemann/zeptoforth/pull/60#issuecomment-1690529766, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACYAPYGPM3RDWHFTQXO6O3XWZLRRANCNFSM6AAAAAA3XUXXMQ . You are receiving this because you authored the thread.Message ID: @.***>

[tabemann]

I'm not sure of a good answer for that ... is pySerial normally installed via a debian package or manually? I'm actually surprised venv is a separate package, I thought that was a core part of python now. Either way, adding a fallback or flag/env variable to make the venv stuff optional wouldn't be too difficult. The other nice thing about venv's is that if something ever goes wrong with it, the right answer is almost always delete the venv and rebuild it. We use a technique like this all the time at the day job.

I installed pySerial via Debian package, but that was a while ago, and I don't see any reason to not install it via PIP. As for why venv is a separate package, take into consideration the love of the Debian people to divide up things into separate packages as far as possible.

All in all, I think in retrospect that doing this with venv was the right way to do it, because of its reducing dependency issues by limiting packages to a particular project (the whole Haskell Cabal hell mess reminds me of how making packages project-specific is a good idea).