Github Release Action for adding binary executables to Release Assets

JensHeilig commented 1 year ago

Is your feature request related to a problem? Please describe

Some (many?) users cannot run python scripts directly on their computers. Python may not be installed by default (Windows) or they are running Docker containers where python is not available (example: openHAB). Even though it is not hard to run pyinstaller --onefile decode-config.py to create an executable, it would be nice if executables for some of the most popular platforms (Windows, Ubuntu 20/22, MacOS 10/11-) were available for direct download from the Github Releases page.

Describe the solution you'd like

A GitHub action which builds executables for the platforms listed above and adds them to "Assets" on the Github Releases page, in addition to the already present source code archives.

Describe alternatives you've considered

Building locally each time I install a new (major) version of Tasmota. And forgetting how to do that exactly each time I try (Yes, I know that this is a me-problem)

Additional context

Since this is a me-problem with probably low priority for most people I will try to create a PR containing a Github Action to implement this feature request. It has been mentioned before here (and not by me), so I assume there is at least some interest in the user-base. The PR should be fairly low-impact on the project, except for polluting the "Assets" panel with a larger number of choices for people to select from.

curzon01 commented 1 year ago

Did you test the resulted bins on the different pyhsical OS? In my experience pyinstaller is not cross-platform capable, the binaries do not run on each OS

JensHeilig commented 1 year ago

I tested some of the binaries:

windows binary runs on my Windows 10 machine (21H2)
MacOS 10.15 binary runs on my Mac mini with MacOS 10.13
Ubuntu 20.04 runs on my Debian-11 (Openmediavault-6) machine. Will test this one on a couple of other Ubuntu machines.

I have no access to Windows-11, Ubuntu-22.04 (but I can test the latter in WSL in the next days) or a newer Mac for testing.

But the binaries for e.g. Ubuntu 22.04 will not run on Ubuntu-20.04. Had to learn that the hard way. That is why the filenames list very specific OS versions to run on.

curzon01 commented 1 year ago

Test real cross-creation:

Test the binaries created under Windows on Ubuntu and Mac too
Test the binaries created under Ubuntu on Windows and Mac
Test the binaries created under Mac on Windows and Ubuntu

My experience is that this doesn't work 'cause pyinstaller can't created runable cross-binaries

JensHeilig commented 1 year ago

I‘m not asking for cross builds in this FR. AFAIK you are correct, pyinstaller cannot create cross-binaries. An it does not need to either.

What I did in PR #64 is to use Github‘s build infrastructure to build on the platforms available there. No cross-compilation is involved or required.

The result can be seen here: decode-config_buildAssets_Test

curzon01 commented 1 year ago

Ok I see actions installs differnet OS when creating the binary. pls can you test on Win11, Debian and current Ubuntu 22? Will check mac bin from your test repo

JensHeilig commented 1 year ago

Tested ˋdecode-config —versionˋ on:

Windows 11: Works
Ubuntu 22.04: Works (both, decode-config-ubuntu22.04 and -20.04)
Ubuntu 20.04: Works (only decode-config-ubuntu20.04)
Debian 11 (bullseye): works (only decode-config-ubuntu20.04)
Debian 10 (buster): fail
Ubuntu-18.04: fail

curzon01 commented 1 year ago

looks good - due to the -x flags for the X/OS bin can we use zip at all for Mac and Linux? (tar.xz might be modern but isn't very common - e. g. Linux Mint desk has no default assoc with this extension, zip is assigned everywhere)

Just wonder why we need OS version related bin at all.

Before v11.0.0 (e.g. v10.1.0 Noelle) I created the bins manually based on then /build scripts but Linux and Mac worked on most common OS versions. As I saw you run pyinstaller out of the box, right? Pls have a look into the prev spec and batch files I used to created my onefile

curzon01 commented 1 year ago

your oneliner pyinstaller Mac bin10.15 runs on Mac 10.5 but Mac bin 11 failed on Mac10.5. Currently do not have access to a Mac11

JensHeilig commented 1 year ago

due to the -x flags for the X/OS bin can we use zip at all for Mac (...)

I believe we can: In my tests, the executable bit was preserved when zipping and unzipping a file. This appears to be confirmed here: https://apple.stackexchange.com/questions/329474/how-does-macos-store-unix-file-permissions-in-zip-files

and Linux? (tar.xz might be modern but isn't very common - e. g. Linux Mint desk has no default assoc with this extension, zip is assigned everywhere)

Rather than use zip for Linux I would rather use tar.bz2 or tar.gz. Both should be available in any Linux distribution of the last 10-20 years. And IMHO this is more "natural" to a Linux system than zip (purely subjective, though). Your are correct, xz really is not as ubiquitous as I assumed: In some (perhaps all? Can't remember) of the Docker images I used for testing (Ubuntu:22.04, Ubuntu:20.04, Debian:10 and Debian:11) xz-utils was not installed by default. So, yes: using tar.xz to distribute the executables was a bad choice. I will change this tar.gz and update the PR. Another argument for tar(.gz): It will definitely preserve the executable bit. Zip probably does as well (tested with zip in RaspberryOS), but I'm not sure how universally available this is.

Just wonder why we need OS version related bin at all.

In Debian-11 this error message is printed when running decode-config built for Ubuntu-22.04: Error loading Python lib '/tmp/_MEI5itFJ2/libpython3.10.so.1.0': dlopen: /lib/x86_64-linux-gnu/libm.so.6: version 'GLIBC_2.35' not found (required by /tmp/_MEI5itFJ2/libpython3.10.so.1.0)

Running the package built in Ubuntu-20.04 in Ubuntu-22.04 works, so we could just build one package for Linux. But since this only runs on Ubuntu-20 or later (I tested: it does not work in Ubuntu-18) and Debian-11 (bullseye) we should add the target distribution name so people with Arch, Mint, Suse, ... can see they need to download and run the python file rather than the packaged binary. And since Github only supports Ubuntu-20 and Ubuntu-22 with their hosted infrastructure (Ubuntu-18 is still available but deprecated) I chose to generate two files with the exact OS-name rather than trying to find a filename that everybody would recognize as "both, Ubuntu-20 and Ubuntu-22"

In MacOS 10.14 there is a similar error message when running the binary for MacOS 11:

[83877] Error loading Python lib '/var/folders/98/7kr_yryn7szdk6h9sylkk9pr0000gn/T/_MEI95btfB/libpython3.10.dylib': dlopen: dlopen(/var/folders/98/7kr_yryn7szdk6h9sylkk9pr0000gn/T/_MEI95btfB/libpython3.10.dylib, 10): Symbol not found: _preadv
  Referenced from: /var/folders/98/7kr_yryn7szdk6h9sylkk9pr0000gn/T/_MEI95btfB/libpython3.10.dylib (which was built for Mac OS X 11.7)
  Expected in: /usr/lib/libSystem.B.dylib
 in /var/folders/98/7kr_yryn7szdk6h9sylkk9pr0000gn/T/_MEI95btfB/libpython3.10.dylib

Since the build for MacOS 11 should also run in MacOS 12, and I can test neither, I decided to build only one package for both "newer" OSes. But "OS X 10.x" is deprecated in Github's infrastructure, which is why I decided to build an extra package for this (rather than using it as a "universal" package for MacOS 10-12).

Concluding:

Instead of using packages for Ubuntu-20.04 and Ubuntu-22.04 it is possible to provide only a single package "decode-config-ubuntu-20_22" built on Ubuntu-20.04. If you prefer this, I can easily change that in the GitHub action file.
Instead of using packages for "MacOS X 10.x" and "MacOS 11" it is probably possible (can't test this) to provide only a single binary built on MacOS 10.15. But this architecture is deprecated in Github. If you prefer this, I can easily change that in the GitHub action file.

Pls have a look into the prev spec and batch files I used to created my onefile

Thanks! I wasn't aware of this! Will look into it! When looking though the pyinstaller documentation (I have not used pyinstaller before this) I did not find much towards multi-OS support.

The only helpful switches/spec-file commands were: --icon <FILE.ico> for adding a file to the Windows executable (should not work in MacOS since we have an executable only, not a "real" Mac App). I will add this to the Windows build.

and

--target-ARCH for building x86_64, arm64 or universal binaries. This is only valid for MacOS builds (and probably not necessary, since Rosetta vs. M1 performance should not matter for this application). When I get access to an M1 Mac next week I will test this switch for the MacOS 11 build.

JensHeilig commented 1 year ago

Tested a build with using spec-files (instead of commandline arguments) => works Currently the spec files are all „default“: what pyinstaller spits out if started with commandline argument `—onefileˋ. But when setting platform-specific options this can now be done in the spec files instead of the github action workflow file.

tested the ˋ—target-archˋ switch on MacOS 11: build fails with some „not supported“ error. Since there is little benefit in using this switch I will not pursue this further.

JensHeilig commented 1 year ago

Additional test: The binary for MacOS 11 runs on a MacBookAir M1 with MacOS 13.

curzon01 commented 1 year ago

That's fine, thx a lot - will check PR these days (a lot other on table) but what I've already seen is need change of dir name build to build-executables or someting different because 'build' is already used by pypi package creation action and build will be completely destroyed each pypi run

curzon01 commented 1 year ago

Lets go with next release

tasmota / decode-config