Feature Request: Providing prebuilt binarys

[KwadFan]

Hi,

It would be good to have prebuilt binarys, possibly static linked, with as few dependencies as possible.

This should be done with the following in mind.

Having different Branches for development and Release. As an example:

main/master - Release Branch. develop - "nightly" branch.

The idea behind that is you could provide stable packages with the main/master branch, but for testing it might be better to compile on the host instead using prebuilt images.

Supported architectures should be: x86(_x64), armhf ( Raspi ), arm64 ( Raspi and armbian ) and armel. This would cover the most use cases.

[ayufan]

I plan the following:

• Release .deb that as dependencies include what needs to be installed - such will be auto installed via apt install ./file.deb
• Release .deb for buster and bullseye due to different packages provided
• Release amd64, armhf, arm64: this supports all since RPI2 and RPI0.2W
• Release variant targeted for Raspberry PI (that depends on libcamera [support for modern cameras]) and other SBC (does not implement libcamera)
• currently, I do not plan to release armel for RPI0: it is extremely slow for that usage anyway

As for branches:

• The master is always supposed to be stable
• Use feature branches for testing new features
• Use semantic versioning
• Every some time there will be release either marked as pre-release (beta) or release (stable)
• For best stability everyone should install release

[KwadFan]

Releasing as a real deb makes more work as needed here. Also keep in mind that these packages has to be GPG signed to be accepted by apt without setting unsecure trust level for your Repo.

It is enough to provide an list of dependencies and a "tar.gz" or "tar.xz" for the binary in my opinion.

What I meant with architectures was targetting not only RPis ! Those are waaayy to expensive these days. It should support a wide range of SBC and "normal" x86 machines.

This includes unfortunatly that not always a Mali could be used to encode streams I think. Having a fallback mode where the cpu is doing at least an MJPG stream is enough.

[ayufan]

This includes unfortunatly that not always a Mali could be used to encode streams I think.

Mali is a GPU, it is not used for decoding/encoding at all. All SBC do have a dedicated VPU units for this purpose.

Also keep in mind that these packages has to be GPG signed to be accepted by apt without setting unsecure trust level for your Repo.

Not true. The simplest is - and it will ensure that all dependencies required are installed as well:

wget .../<file>.deb
apt install $PWD/<file>.deb

and "normal" x86 machines

I don't think anyone those days should be using x86, the amd64 is pretty widespread.

[KwadFan]

wget .../<file>.deb
apt install $PWD/<file>.deb

Yes, for Installing and now please explain how to upgrade the binarys using moonrakers update manager .... So it has to be in your sources.list at least. And that requires GPG Signed packages...

and "normal" x86 machines

I don't think anyone those days should be using x86, the amd64 is pretty widespread.

Thats what I meant with x86, I dont mean i686 those machines are reare those days I think ;)

[ayufan]

Yes, for Installing and now please explain how to upgrade the binarys using moonrakers update manager .... So it has to bin in your sources.list at least. And that requires GPG Signed packages...

I assume that we would install this with update of crowsdnet. So, crowsdnet would either bundle package or pull this from the internet. Since it is crowsdnet dependency to install and configure the web streaming.

This is why I also assume that it would be good for crowsdnet to ensure that ArduCAM libraries are installed if requested by user.

However, if you prefer apt.conf I do publish all my debian packages fetched from GitHub releases and properly signed with GPG keys: https://deb.ayufan.eu/. This repository is not exposed at this moment as it does not publish anything yet.

• Publish .tar.gz is clumsy if the tool depends on libraries of the current system, and it can break due to mismatched libraries
• Publishing statically compiled is not ideal either as it would link libcamera.so (and this is provided by RaspberryPI and ArduCAM separately and they differ)

So, if someone wants to install from dependencies directly it should compile directly on a target system. Otherwise a .deb should be used created against existing libraries of the system on which it will be installed, this is why distinction between RPI vs non-RPI and Buster vs Bullseye.

[KwadFan]

I assume that we would install this with update of crowsdnet. So, crowsdnet would either bundle package or pull this from the internet. Since it is crowsdnet dependency to install and configure the web streaming.

And thats the Issue here. Crowsnest it self has no abilitys to upgrade any packages. that does moonraker. Crowsnest is update due moonraker via a simple git pull and has no option to run custom scripts.

But providing .deb isnt your Job here, these are system dependend. Therefor Job of OS mainatiners, imo.

Otherwise you have to provide packages for every single Distro that might be used. At least Ubuntu, Debian Buster/Bullseye.

Publishing statically compiled is not ideal either as it would link libcamera.so (and this is provided by RaspberryPI and ArduCAM separately and they differ)

As far as possible. You know I no not much about C/C++ here :) What comes to my mind was an approach like go does. Binary runs and need nothing as long as compiled for matching architecture.

[KwadFan]

But that said, I dont want to disturb your plans on providing debs.

For me it would be much easier to install dependencies with the crowsnest installer and once in a while grabing a "tarball", which could somehow implemented in crowsnest.

[ayufan]

Otherwise you have to provide packages for every single Distro that might be used. At least Ubuntu, Debian Buster/Bullseye.

My experience says that it is good enough in here to provide debian support (for something like this) and it should be good enough to install on all other Debian derivatives.

Crowsnest it self has no abilitys to upgrade any packages

How it runs at as a service then? Something needs to create a proper systemd service :)

I might not understand something, but this already compiles some apps: https://github.com/mainsail-crew/crowsnest/blob/master/bin/Makefile.

It does mean that least from this that [git_repo] definition misses syntax to re-install service after update.

[ayufan]

Anyway, lets work on this and see where we end-up. I just consider .deb a safe way to distribute application with system-level dependencies that should not be statically linked.

I do see that at least (via Moonraker setup) the Cmnd_Alias APT = /usr/bin/apt-get is allowed via sudo, so crowsnest should be able to do it.

[KwadFan]

Yes, and no.

Crowsnest runs as pi, which is not able to use sudo commands without a password. That would require to allow that what I never will do. The reason is simple. Security. Moonraker for example uses Polkit. But i dont want to mess with polkit in a bash script that runs a binary.

[KwadFan]

The Makefile is only for install.

Reinstalling services is not needed. Once the service file is in place you dont have to change that.

[ayufan]

Reinstalling services is not needed. Once the service file is in place you dont have to change that. Crowsnest runs as pi, which is not able to use sudo commands without a password. That would require to allow that what I never will do.

There are two things:

• the service itself runs as pi: https://github.com/mainsail-crew/crowsnest/blob/master/file_templates/crowsnest.service#L25 which is great idea since the pi has by default video group
• the [git_repo] runs as a moonraker user so it has access to apt-get and alike

I think in general it should be make install to be re-executed (or have make update) run by [git_repo] to allow updating things like package or binary camera streamer, or whatever else is required. This would allow to:

• ensure that correct dependencies and binaries are installed
• run sudo systemctl daemon-reload and sudo systemctl restart ... to automatically update and restart crowsdnet without having to restart system (I did not test this, but I assume that this is how it works)
• it does not change the permission model with pi

This allows to adopt things over time, add or remove existing dependencies seamlessly for user.

[KwadFan]

As already said, moonraker does NOT allow to run any scripts or commands!

I think you are mixing something up here. Service files should not be changed by runtime! You should really install crowsnest on a testmachine to see how this whole thing works. The service file is only there to start crowsnest main script. Nothing more. Any updates that are done are a simple git pull.

Moonraker does only restart the service, which ends up restarting the bash script.

This allows to adopt things over time, add or remove existing dependencies seamlessly for user.

But this will never happen with moonraker.

[ayufan]

I think you are mixing something up here. Service files should not be changed by runtime!

Of course. It should be updated by the update process.

As already said, moonraker does NOT allow to run any scripts or commands!

So you are saying that this does/will not work, thus will not run scripts/install.sh and only parse for packages?

[update_manager update_manager]
type: git_repo
path: ~/crowsnest
origin: https://github.com/mainsail-crew/crowsnest.git

# a new bits that are added where it makes Moonraker to be aware of crowsnet being a service
install_script: scrips/install.sh
is_system_service: True # unsure if this is needed
managed_services: crowsnest

I guess TIL and this is surprising development as it really makes updates harder and does not really improve security.

However, if we add a proper debian repository that is signed it will nicely work with the Moonraker to automatically update the package where needed. Without that I don't see any way to update camera-streamer at all as binaries will not be self downloaded.

[KwadFan]

Of course. It should be updated by the update process.

For what reason? If the "binary" changes its behavior maybe, but thats the point. In our "Klipperversum" it shouldnt change.

So you are saying that this does/will not work, thus will not run scripts/install.sh and only parse for packages?

Unfortunatly yes, this isnt wanted by Arksine due potential security risks. If it would be executed you could enhance "functionalitys" that arent wanted on an network device ;)

Adding a proper apt source wouldnt be a problem, which leads to that what I said, it needs to be signed. But so much effort worth it? Wouldnt it be easier to abuse moonrakers zip update function? Thats how mainsail gui gets updates. Requires no signing and building debs. Only a zip containing the binary and maybe needed .so ;)