search

createcandle / Candle-manager-addon

An add-on for the Candle Controller / WebThings Gateway. It allows you to upload code to Arduino's with the click of a button.
MIT License
5 stars 5 forks source link

Cross-compile rather than bootstrap #12

Closed mrstegeman closed 4 years ago

mrstegeman commented 4 years ago

The addon-builder now has support for cross-compiling Python add-ons for various architectures and Python versions. It would be helpful to use it rather than bootstrapping on target in the background for a (long) time before being able to actually start up. You could even selectively include the property arduino-cli binary in your package, to make it smaller. I've done this for the Eufy add-on, as an example.

createcandle commented 4 years ago

I'm not sure what rather than bootstrapping on target in the background for a (long) time before being able to actually start up means.

You could even selectively include the property arduino-cli binary in your package, to make it smaller. I've done this for the Eufy add-on, as an example.

Would that mean I would have to create files for each architecture individually? And then upload them as releases on Github? That sounds like a lot of work.

mrstegeman commented 4 years ago

Previously, you were using bootstrap.py to download and build all of the Python dependencies on the Raspberry Pi, after initially installing the add-on. This can take a long time, and it's unclear to the user what's happening.

Recently, I added support for Python add-ons to our addon-builder, which is what we've always used to cross-compile our Node.js add-ons for different architectures. Since this add-on has native dependencies that are compiled for a specific target, the addon-builder will handle building the entire matrix of Python version and architecture pairs, e.g. linux-arm for Python 3.7, linux-arm64 for Python 3.5, etc.

The process is this:

  1. I add this add-on as a submodule for the builder.
  2. Each time you want a new release built, you file a PR in the addon-list repo (as usual) and let me know.
  3. I kick off the builder. It automatically pulls the latest code from your GitHub repo and runs the package.sh script across the matrix.
  4. I update the URLs and checksums in your add-on list entry in your PR and we're done.
createcandle commented 4 years ago

I don't know anything about what the addon-builder is, how it works, why I would want to use it, etc. The addon builder github doesn't really help me understand. So for now I don't think this is a direction I want to take this addon in.

mrstegeman commented 4 years ago

For a Python add-on with native dependencies, in order to run on everything we support, you have to cross-compile your add-on for the following targets:

It's a similar story for Node.js add-ons, although we only build for 3 versions of Node.

The addon-builder does this by using custom Docker-based toolchains (available here), which are run by this workflow, using this script and this script.

You may think this is unnecessary, if you only want to support the Raspberry Pi, but even with that, users could run different distributions, different versions of Python, etc.

If you are still resistant to using it, that's fine. However, you can still keep these changes in case you change your mind in the future. You just run the script like this now:

ADDON_ARCH=linux-arm ./package.sh
flatsiedatsie commented 4 years ago 
I add this add-on as a submodule for the builder.
Each time you want a new release built, you file a PR in the addon-list repo (as usual) and let me know.
I kick off the builder. It automatically pulls the latest code from your GitHub repo and runs the package.sh script across the matrix.
I update the URLs and checksums in your add-on list entry in your PR and we're done.

I must have overlooked this somehow. That sounds pretty good. And more secure too, since you can be sure that the Github code is what's in the addon.

flatsiedatsie commented 4 years ago

How do I then do local tests first? Do I keep a separate 'old school' package-oldschool.sh for that?

And can I still have files in the github that don't make it into the final 'tarball'? How do I indicate which files actually matter? Is that why the package.json file is still around?

mrstegeman commented 4 years ago

That sounds pretty good. And more secure too, since you can be sure that the Github code is what's in the addon.

I agree. 😄

How do I then do local tests first? Do I keep a separate 'old school' package-oldschool.sh for that?

You can still use the same package.sh script. Just run it like this:

ADDON_ARCH=linux-arm ./package.sh

And can I still have files in the github that don't make it into the final 'tarball'? How do I indicate which files actually matter? Is that why the package.json file is still around?

Your package.sh script dictate what goes into the tarball. The addon-builder uses your package.sh script. package.json is still around to support gateway versions <= 0.9. If you don't care about those anymore, you can get rid of it.

flatsiedatsie commented 4 years ago

I actually tried to remove package.json recently, and it lead to a failure. The github script that checks new addon uploads still requires this file to be present.

I noticed in the package.json code you provided you are still adding files to the package.json? Is that just for legacy purposes? If so, can I remove that code and remove the package.json? (assuming the server block is lifted)

flatsiedatsie commented 4 years ago

One more thing: the script currently deletes files in the base arduino-cli folder. I'd like it to not do that :-) Could you modify it to only delete files in package directory once it's created?

mrstegeman commented 4 years ago

I actually tried to remove package.json recently, and it lead to a failure. The github script that checks new addon uploads still requires this file to be present.

You should know how to fix that now. 😃

I noticed in the package.json code you provided you are still adding files to the package.json? Is that just for legacy purposes? If so, can I remove that code and remove the package.json? (assuming the server block is lifted)

Yes, for 0.9.X and below, we actually checked that files array at run time.

mrstegeman commented 4 years ago

One more thing: the script currently deletes files in the base arduino-cli folder. I'd like it to not do that :-) Could you modify it to only delete files in package directory once it's created?

No problem, PR updated.

createcandle commented 4 years ago

thanks! done.