Mention packaging in GNU Guix

[yewscion]

Hello,

CBQN is now packaged for GNU Guix.

It might be worthwhile to add a note saying as much to README.md, to spread awareness that manual compilation is unnecessary if a user wants to use CBQN.

If a user has GNU Guix installed, they only need to run the following command to install and then use CBQN, eliminating all other setup needed by the user: guix install cbqn. It is currently tested on x86_64 and confirmed working (by me and our CI backends), but not for aarch64 due to an issue with a dependency of librsvg (gdk-pixbuf-query-loaders) failing to build properly (outside of my control and experience).

There is a badge that can be utilized if You prefer; it looks like this: If You'd like to use it, feel free to add it as a file to Your repo so it doesn't move around; It is licensed under GPLv3/FDLv1.3, which is compatible with this project.

[dzaima]

Given that CBQN is also on (at least as far as my knowledge goes) nixpkgs and AUR, I'd want some uniform listing across those, probably just links in some section (also mentioning WinBQN).

The running page currently has this information (including what I guess might be outdated now? "and bqn in development Guix (not 1.3.0)"), but it might be worth moving to here? @mlochbaum

[yewscion]

That makes sense to me, just wanted to report that it is in the distribution now for the most part.

Incidentally, GNU Guix is a rolling release distribution; though there are unstable/user channels You can add, the 'development' version of GNU Guix would probably be some branched-off version that only <10 people use.

The releases are more about OOTB experience than they are about versioned updates a la Ubuntu or Fedora.

I guess what I'm saying is that it's fine to drop the "development" qualifier, if You want.

[mlochbaum]

So presumably you download Guix 1.3.0 and then update the packages? The Guix website completely fails to mention anything about rolling release as far as I can tell.

[mlochbaum]

Here's my understanding of the package situation. Is this correct?

The package has been built for the architectures listed under "Builds:" but has only been tested on x64_64. Installing just downloads the binary but an advanced user would be able to figure out how to build from source with bqn.scm, and possibly get it running on other architectures that way. bqn.scm defines a few packages including cbqn and dbqn, but none of them are called bqn.

[mlochbaum]

Planning to add this table to running.md. I think it has everything a user of one of these OSs would want to know? The Nix and Guix packages bootstrap the bytecode but I wouldn't expect someone downloading to care whether this step has been done by dzaima or the packager.

OS/distro	Package	Type	Singeli
Windows download	WinBQN	Binary	No
Arch AUR	cbqn-git	Source	No
Nix	cbqn	Binary	No
Guix	cbqn	Binary	Yes

[yewscion]

Let me outline a few things, as a person who uses GNU Guix as my daily driver:

• Guix is a source-based, rolling-release distribution of the GNU Operating system, which can (currently) use either the Linux-Libre or Hurd kernels.
• Guix is also a reproducibility-focused package manager built and configured entirely in Guile Scheme, so it can run on any OS (that supports a few prerequisites, such as symlink support and nss support).
• If we are talking about using Guix as an OS: During installation, the image used to boot the installer is frozen at commit a0178d3, so that it is known to be working.
• However, once You've rebooted into the system installed by it (or, for more advanced users, before You begin the installation so You have access to unofficial channels), a critical step is updating the system using guix pull and guix system reconfigure /etc/config.scm. This puts the user at the latest commit, and is recommended here in the manual.
• So, though it isn't mandatory to install any updates (and indeed, a user can pin each channel to a specific commit so it is never updated), it is expected for the average user to be running the latest commit (There is actually a warning that comes up if the user doesn't run guix pull for 10+ days each time they use the guix command, too).
• If we are talking about Guix as a package manager, the conclusion is similar, though without the reliance on a boot image: The setup script lives in the repo and is expected to be updated afterwards in the same way.
• As for the builds: cbqn is marked as being available for both both aarch64 and x86_64, however it currently only usable on x86_64 because a package that a package that a package that cbqn relies on (gdk-pixbuf-query-loaders) is failing to build. Once that is fixed, it will then (presumably) compile correctly, and therefore be usable on both systems.
• GNU Guix uses an optional binary substitution server to ease the costs of compilation for its users. This is possible because all of the builds in the system are meant to be reproducible, meaning they result in identical binaries each time regardless of the state of the system on which it was compiled (or when it was compiled, etc). However, the main use of GNU Guix is source-based, as should a substitute be unavailable (whether because the user disabled it during use or installation, the substitute server is down or too slow, or the CI systems have not gotten around to rebuilding the most recent version of that package yet) building from source is an automatic and default fallback.
• So, defining GNU Guix as a 'binary' release is a bit erroneous: While there are two official substitute servers that offer some of the packages as prebuilt archives in order to allow people with netbooks from compiling mozilla browsers and texlive from scratch each time those packages update, this is optional from the start and not used if it is slow or unavailable. And if it is use, the binary archives it pulls are checked to correspond 1:1 with the binary that would have been produced by a source compilation. Very few packages fail this check for avoidable reasons; more information here
• I dunno how You want to write down such a nuanced classification (I've heard Nix is similar though I've never used it), but perhaps 'Source/Binary' or 'Both' would be better than solely listing Binary?

[yewscion]

@mlochbaum That looks fine to me, aside from the Binary/Source concern in my last two bulletpoints above.

[mlochbaum]

Yeah, it's pretty much the same as Nix, and if I had gone with that background knowledge I wouldn't have been confused. The website is very misleading with how it keeps saying version 1.3.0 everywhere (it doesn't matter!) and describes it as a "standalone Guix System" when the thing you actually care about are the latest packages, which won't be included.

[mlochbaum]

Wait, is there only one build for generic x86-64? The Singeli code is only used if AVX2 is enabled at compile time, so it's kind of useless if that's the case.

[dzaima]

Wait, is there only one build for generic x86-64? The Singeli code is only used if AVX2 is enabled at compile time, so it's kind of useless if that's the case.

Been trying to understand that right now too; the o3n-singeli target includes -march=native (which is fine for a local build, but definitely not appropriate for a downloadable binary release), and singeli means it requires the x86-64 extension AVX2 (i.e. won't compile on older CPUs, or non-x86-64 ones); that is to say, o3n-singeli can only be used for a local build on the CPU you'll run it on, and only on modern x86-64; unless guix can specialize for that, staying on o3 is best for now (though it will come with losing many optimizations)

[yewscion]

Interesting, I was unaware of these particulars when I packaged CBQN before. Reaching out to see if I can get some advice; Might submit a patch to fall back to o3. Is o3-shared alright to use in this case, or do the AVX2 extensions impact that as well?

[yewscion]

Will be amending the package to default to o3, but will also add a separate transformation for o3-singeli compatible machines. Are there other targets that might be useful in specific situations? I can likely add them with the same patch… But I'm pretty unfamiliar with some of the referenced options (like I was with AVX2, my bad, there).

[yewscion]

In particular, perhaps the targets should be the single-* varieties instead? The README seems to suggest it may be more optimized. I'm also unfamiliar with -g.

[mlochbaum]

The other target we're using is BMI2, and we're likely to use PCLMUL and FMA3 in the future. All of these are included in most CPUs with AVX2. I think -march=haswell (maybe with -mtune=icelake-client?) is the setting to use.

-march=native queries the architecture it's currently being built on, which means the fact that you're able to get reproducible builds out of it is confusing. I guess Guix probably overrides this somehow?

The improvement with single-* is going to be pretty minor if it's there. But then the reason not to use it is that it doesn't allow incremental builds, which you're not going to do. I guess I'd say to not use it just to get a more "standard" build.

[dzaima]

o3-shared is a generic build, okay for binary releases; -g is only for debugging purposes; single-* ideally shouldn't give many performance benefits, I should re-check and update the readme (they're largely there for hypothetical situations where the makefile or incremental builds don't want to work).

o3-singeli isn't a valid configuration currently, though make singeli=1 f='-mavx2 -mbmi2 -O3' c is an alternative that does that (i.e. targeting & requiring x86_64+AVX2+BMI2, i.e. all current optimization targets; it appears that AVX2 without BMI2 currently doesn't compile), without being just -march=haswell which presumably doesn't go well on AMD CPUs, but of course would stop being optimal when FMA3 or PCLMUL are targets.

pre-compiled binaries and x86-64 extensions don't go together (and the alternative of runtime architecture detection probably won't come for some time either).

[yewscion]

Yeah, I think the override might have to do with -mtune, though I haven't looked for this in the source.

Copy that, maybe I should submit a patch to include the following packages, then:

• cbqn: Will use o3, and be the default package to install for cbqn.
• cbqn-singeli: Will use make singeli=1 f='-mavx2 -mbmi2 -O3' c, and will be noted as not for older CPUs and not for 'FMA3' or 'PCLMUL'.
• cbqn-haswell: Will use make -f='-march=haswell', and will be noted as being for modern AMD CPUs and for 'FMA3' and 'PCLMUL'.

All three will have a lib target in addition to the default out target, so the user to install the shared library should they desire. For now, this will use o3-shared, but the hooks will be there for future optimization should the shared library fragment similarly to the binary.

As for the single-* builds, it looks like they are analogous to the non-single anyway, perhaps I could define a single target for each package that uses single-o3 or single-c as appropriate, including the manual flags mentioned above?

[dzaima]

haswell is for modern intel CPUs (i.e. 4th generation or later, which is from 2013), but I don't think it's worth having it when the more generic -mavx2 -mbmi2 is an option; FMA3 and PCLMUL shouldn't be mentioned, they currently aren't used by CBQN, and it isn't even a given that by the time they are they'll be checked at compile-time anyway.

single-* builds are too much hassle for insignificant gain; they're not exactly analogous, and don't support Singeli (as it generates static functions with equivalent names, requiring them to be across multiple compilation units).

[yewscion]

Copy that, then, how about one package with the following outputs, then:

• out: The default, what happens when a 'user installs cbqn' using guix install cbqn. Uses make o3.
• lib: The shared library, using make o3-shared, as it is generic.
• singeli: The 'extended' version, closest to what is in Guix now, that has to be specifically installed by a user selecting it (similar to the way Guix handles JRE vs JDK variants of packages). Uses make singeli=1 f='-mavx2 -mbmi2 -O3' c.

[yewscion]

For Your Awareness, Patch submitted: https://issues.guix.gnu.org/58100