Use ql2nix in nixpkgs

[SquircleSpace]

nixpkgs.lispPackages needs some help. I've got a plan, but I need a bit of help integrating it into nixpkgs.

A few years ago, when I was first getting into Nix, I tried to make lispPackages better. I wanted every quicklisp package to be included. I wanted every package to be pre-compiled and ready to load. I wanted a haskellPackaged-style experience of creating a compiler environment in which the desired packages (and only the desired packages!) were accessible.

The lisp ecosystem as it exists today is very much incompatible with those goals. As you're probably aware, arbitrary compile-time logic is an essential feature of Common Lisp and ASDF system definitions are just Common Lisp code. The end result is that a lot of packages do nasty things. For example, some packages use an (asdf-discouraged, but still common) pattern where they check if some other system has been loaded and then behave differently. That makes reproducible builds very difficult and sharing of compilation artifacts even more difficult.

I struggled to make it work for so long. I layered hacks on hacks on hacks, gave up, started over, and found myself layering hacks again. Eventually, I gave up. I added the packages I needed to nixpkgs and left shaking my head. It wasn't meant to be.

I'm back, and I've got a new plan! I decided to stop trying to copy haskellPackages and to take a bit of inspiration from how the node ecosystem works. Okay, I admit, I know almost nothing about how the node ecosystem works with Nix. I just barely dipped my toes in the waters when I realized that I had been approaching the lispPackages problem all wrong. I threw together a little project called ql2nix. Its still quite rough, but it seems to be working. I haven't tested it on some known-bad packages, but things are looking good so far.

I want to give up on lispPackages as we know it today. Instead of trying to make Nix into an alternative to Quicklisp, Nix will leverage Quicklisp similarly to how Nix leverages NPM. Here's the sketch for how it will work.

1. Using ql2nix, you create a Nix manifest of the Quicklisp releases (i.e. source tarballs) and systems (i.e. ASDF systems) that are required for your project. ql2nix pulls that info from your local Quicklisp installation. So, its just freezing the dependencies you're already using. The neat thing is this naturally supports alternative Quicklisp dists (i.e. package repositories) that are starting to spring up!
2. Using the Nix expressions in ql2nix, you can then create a Quicklisp dist that contains all and only the systems that were captured in the closure earlier. You can install this dist into any Quicklisp installation! After that, ql:quickload and such will install using the dependencies captured in step 1. I'm calling this dist "nixlisp".
3. Next, using a bit more Nix embedded in ql2nix, we can use the nixlisp dist and ql:bundle-systems to create a folder containing the systems captured in step 1. Quicklisp helpfully includes a bundle.lisp file that can be loaded to set up ASDF so that it can find the systems contained within the bundle. bundle.lisp doesn't depend on Quicklisp. Now we've got a simple file that, when loaded, gives us access to all the dependencies that were captured in step 1... except we're not dependent on Quicklisp and the dependencies are contained within the Nix store!

This will run into problems if a system tries to create files in the source directory. I know some systems do that. We can probably get those systems to store their files in ASDF's output directory (~/.cache/common-lisp/... by default) using pull requests or patches.

So... what is left to do? Well, I don't think nixpkgs NEEDS to do anything. I think ql2nix is completely usable on its own without any nixpkgs support. Personally, I'm going to start migrating my personal projects away from using lispPackages. I'm filing this issue because I think you might find it a bit easier to build some Common Lisp programs using ql2nix. For example, stumpwm might be easier to maintain if it isn't dependent on keeping lispPackages up to date. I noticed stumpwm was broken for a long while about ~1 year ago. It seemed like the issue stemmed from lispPackages being difficult. I also think that ql2nix could use a bit of help from someone that knows how to Nix better than I do. Am I doing dependency injection the right way? Is the Nix output of ql2nix reasonable? Stuff like that.

[SquircleSpace]

I forgot to link to ql2nix! https://github.com/bradleyjensen/ql2nix

[7c6f434c]

I think the problem with StumpWM was that a proper and customisable build of a binary build was just not there. I think the most popular packages are more likely to have a saner build system, so I guess I could bump lispPackages from time to time (and there is a chance that our approach would allow to eventually provide most of the functionality that Quickref now does weirdly), but indeed a good fully-ql-based approach is very welcome!

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[bqv]

I'm still interested in this, since lispPackages is pretty woeful