The Road to 1.0

[staticfloat]

This issue will serve as a roadmap of sorts, a kind of grouping of "accepted feature requests" generated by myself, Mosè, and other concerned parties. While BinaryBuilder.jl already does a fantastic job for many pieces of software, there are things that make it unnecessarily difficult to work with, or that limit it in some ways, and we should address these things as best we can.

BinaryBuilder.jl 1.0 should, in my mind, be ready to put virtually all other ways of compiling software out of business. It should work flawlessly for the Julia ecosystem, but also be usable for other groups as well. It should be feature-complete and well-tested, and have as few hidden performance or and usability warts as possible. We're a good ways toward fulfilling this vision, but I see the following areas as needing improvement before we can truly wash our hands and say that this area is solved:

Build Platform

• Bootstrapping is still a huge issue. For machines that can support it, we should allow emulation of targets within the build environment via qemu-user-static, wine and darling.
  • To do this properly, we'll need a mechanism in BB to check if the user has already loaded the binfmt_misc kernel module and the darling kernel module, and prompt them to load it if not.
  • I'm as-yet unaware of any FreeBSD emulation software.
  • Auditing is slow; it should be parallelized. I'd like to see a workqueue/threadpool kind of approach.
  • Feedback/logging from BB exists, but it's not great. It would be nice to have a structured output that can be consumed by tools after the fact, as well as a better categorization of feedback so that we can ignore certain classes of warnings more easily.

Build API

• We need the ability to produce multiple outputs from a single build, which will entail a small overhaul of the build_tarballs() API. I refer to this as the BuildMetadata rework, and it is best summarized in this comment,

JLLs

• We need the ability to for JLLs to serve multiple "variants" of themselves. With the advent of CompilePreferences and Pkg hooks, it's possible this can be done by having debug versions of a package listed as debug=true tags in their platform objects, then doing a runtime lookup of a compiletime preference to see if debug has been set to true and selecting those artifacts instead.
• With CompilePreferences we could do away with Overrides.toml and instead be much more flexible: we could have an overrides dict within the preferences itself that selectively overrides the paths of individual products.

Other

• We should continue to split out pieces of BB/BBB functionality into separate packages. The sandboxing and runner stuff would make a good package, IMO.
  • Potential name: Sandbox.jl, with backing UserNS and Docker runners contained within it.

This will be a living document and get updated as more good ideas come to the foreground.

[PallHaraldsson]

For machines that can support it, we should allow emulation of targets within the build environment via qemu-user-static, wine and darling.

Would this be reliable, where e.g. wine supported? Maybe I'm not up-to-speed on BinaryBuilder, and while I might use wine personally, and it's probably even pretty compatible by now, it's not perfect. Would the result be compromized and some virtual machine with real Windows be better?

[giordano]

Would this be reliable, where e.g. wine supported? Maybe I'm not up-to-speed on BinaryBuilder, and while I might use wine personally, and it's probably even pretty compatible by now, it's not perfect.

Wine is certainly not perfect, but right now we can't run foreign executables at all.

Would the result be compromized and some virtual machine with real Windows be better?

The whole point of this framework is to not have to have access to the target platform and instead have a single environment to build for all the desired platforms.

[staticfloat]

Mosè and I are discussing how best to deal with complex build scenarios involving bootstrapping and canadian crosses and whatnot.

We envision a possible future where we have commands such as bb target $triplet which mounts the necessary compiler shards and sets up the environment such that /workspace/destdir is a symlink which now points to /workspace/destdir.d/$triplet, CC=$triplet-cc, etc....

This would allow hot-swapping between multiple different builds at a time, which is very useful, and also for more convenient interactive exploration.