Open SimonLSchlee opened 5 years ago
Relevant related work: Docker's image building engine was recently redesigned to use BuildKit, an engine for deterministic parallel caching builds that uses a low-level makefile-like graph to represent build actions. BuildKit has an explicit notion of "builder frontend languages" like Dockerfiles, Makefiles, and other build tool configuration formats and uses a #lang
-like mechanism to convert those kinds of files into BuildKit's representation. Looks like this:
# syntax = docker/dockerfile:1.0
... Dockerfile contents ...
Or for a makefile, like this:
# syntax = someuser/makefile:1.2.3
... a regular Makefile ...
BuildKit uses the # syntax = somedockerimage
comment to run a docker container that processes the build file into BuildKit's standard build graph representation, much like the way Racket uses the #lang somemodule
line to load a module that parses and expands the rest of a source code file.
So @SimonLSchlee, I think you're definitely onto something valuable here. What if we made a docker image that BuildKit could use somehow to bridge into the world of #lang
s?
Also related is the scripty
package, which lets you write shell scripts implemented with Racket that automatically install any packages they need when the script is run. Looks like this:
#!/usr/bin/env racket
#lang scripty
#:dependencies '("base" "somelibrary")
------------------------------------------
#lang racket/base
(require somelibrary)
(do-some-stuff)
(do-some-more-stuff)
(do-other-stuff)
Related: API for Cross-Platform Configuration My guess is, that this is already used somewhere to create the racket builds. But it would be nice to make it easier to use and see an example of its use.
@jackfirth I think there are 2 different cases:
packages which are pure racket code and only indirectly use platform specific code (via other packages)
packages that directly have platform specific code or libraries
I think docker based tooling would be helpful for 2. because it would allow others to understand and update platform specific packages in a reproduce-able way.
For 1. I would prefer cross-compilation that works on the logical racket level.
If the current raco exe
wraps the racket byte code with the platform specific racket, it should be possible to wrap it with rackets that are specific to a different platform, without running that platform. It would be interesting to know if there are special cases where this is difficult to do. Or if it is simply a matter of downloading the platform-specific minimal rackets and some how fuse them with the byte code.
Independent of using Docker or not, BuildKit definitively looks interesting.
I wonder whether docker based approaches would encourage people to create more packages that for example wrap a game engine written in c or rust, together with bindings written in racket.
Maybe there would be a game-engine-lib
game-engine-doc
game-engine-test
and new a game-engine-dev
which allows to build the foreign library from source stick the resulting library into game-engine-lib
, generate updated docs in game-engine-doc
and release new versions of those packages.
I have a pipe dream that WASM might provide an alternative to distributing lots platform-specific variants of native libraries. I have some (and am trying to put together some others), and it is not delightful to create and maintain them, especially as someone who doesn't work with the incantations for building and linking shared libraries on a regular basis (especially on Windows …).
Also relevant is Bazel, the open source version of Google's distributed build tool (internally called "blaze"). That system has a concept of platforms which handles platform-specific dependencies and build actions pretty nicely.
Some time ago I put together a gist with a list of links that might be useful for someone trying to make "multi-platform Racket", in a way that gives individual Racket languages more control over how they behave across different platforms. The gist includes links to lots of articles and specs about Web Assembly, the RISC-V instruction set architecture, Docker images, how Docker handles multi-platform stuff, the Nix package system and OS, how calling conventions are different across different operating systems, and other compiler-related odds and ends.
@SimonLSchlee Your "another possibilty" is reality. The distribution process is implemented by the distro-build
package. The main Racket distribution is a package called "main-distribution". The Northwestern snapshot builds are "Racket plus Tests", because it adds the "main-distribution-test" package. The Utah snapshot site includes "Minimal Racket+GUI+Docs" and "Minimal DrRacket" distributions that are built with smaller sets of packages.
The distro-build
package cooperates with the top-level Makefile
of the Racket GitHub repo to build from scratch. It takes information about how to build as a #lang distro-build/config
program and creates distributions, each for a specified platform and with a set of specific packages. It was specifically designed to support things like course-specific distributions.
@mflatt I am curious what parts of Racket prevent it from being fully cross-compiled. For example why it seems to be necessary to run a windows client machine to create a windows build with distro-build
. I would especially like to know whether racket-cs has less platform dependencies or may even be close to being able to be fully cross-compiled.
I would be interested to help with creating a minimal-racket distro-build
that works completely via cross-compilation, because I think that it would be very attractive for racket to create a "standalone application installer for (gui) applications implemented with racket" for x platforms with as few commands/setup as possible.
Without needing to setup build servers and clients, worrying about virtual-machine-images running windows etc.
If you think this cross-compilation goal is difficult (even for a racket-cs only future), that would also be good to know.
For now I will experiment with distro-build
and virtual box images. But I am unsure how fast I will have results/time for that.
The distro-build
approach starts from sources, including the C sources for the runtime system, so it needs a C compiler. Currently, distro-build
can cross-compile for Windows using MinGW. I don't think there would be any obstacles to cross-compiling Unix variants (not counting Mac OS) with a suitable C cross compiler. Cross-compiling for Mac OS is not doubt possible but more painful.
Meanwhile, as you may know, raco exe
can cross-build (https://docs.racket-lang.org/raco/cross-system.html), as least for the traditional Racket implementation. To do that, you need a built version of Minimal Racket for the target platform. The "Tarball" builds at https://download.racket-lang.org/releases/7.4/ and at the Utah snapshot site are meant to support that. It would be nice to have a raco
tool that fetches tarballs as needed and otherwise makes the process easier. Also, more work is needed on the bundling side to make Racket CS work; cross-build for Racket CS will require a Chez Scheme compiler binary for each combination of host and target platform. Perhaps, with some work, the Racket bootstrap for Chez Scheme could be used to generate a Chez Scheme cross-compiler on demand for any target platform on any host platform.
Putting those together, it would be possible to generalize distro-build
to not build from C sources but instead take advantage of pre-built runtime tarballs (like raco exe
does) to populate the new build. That would be the right thing for creating new Racket distribution variants (i.e., things that you want to have a racket
executable and libraries), but I'm not sure it's the right thing for creating installers for stand-alone applications that are built with Racket. Building on raco exe
and raco distibrute
is more likely the right thing for distributing stand-alone applications.
Point 3 in The Principles of Racket states:
Currently it is possible to create standalone applications by first running
raco exe ...
and thenraco distribute ...
, the end result is a platform specific bundle of the application code as byte code, the racket runtime and other resources and libraries.I propose that we should find a way to make distribution of programs a part of the language (possibly a dsl). So instead of every package having its own ad-hoc method of building release bundles via Makefiles, bash Scripts etc. there is a standardized racket-way of doing that.
-platforms osx,linux,windows
maybe the dsl allows to define all the platform specific parameters you currently give toraco exe
and you don't have to callraco exe
explicitly anymore?raco publish
that has some way of registering user defined hooks/implementations. The idea being that through the hook I can trigger a script, acquire the necessary credentials, possibly via interaction with the user and then proceed to commit and push or upload the release.Node.js's
package.json
allows you to define package specific commands that can be triggered, but maybe there is a more rackety way of internalizing these workflow related things.Another possibility
Maybe the above isn't imagined big enough. Instead of adding more to single packages, we should embrace rackets strengths with multi-collection packages and build another dimension on top of that: meta-packages that describe distributions. Racket and racket-minimal would be two different packages that define what is part of those distributions and there is some command that allows building those distributions from source, or with cached intermediate results and install the distribution or run tests.
This also could make it easier to create use case specific distributions, maybe for a specific racket introductory course with pre configured and customized DrRacket showing the tutorial slides in a split etc.
Are there fundamental flaws in my thinking? Is this a bad idea for some reason?
I think this comment from @jeapostrophe in another issue is relevant to the idea of defining distributions as meta-packages.