Customize ("fixup") Lisp packages before building?

[lukego]

I am looking for a way to customize the set of Lisp package derivations before they are built. Is this supported in the API? Or else how might we support it?

For example I would like to add a failureHook so that broken packages will be "built" into debug information.

I tried to do it like this:

let pkgs = import nixpkgs {};
    baseLispPackages = pkgs.lispPackages_new.sbclPackages;
    updateDerivation = d: d.overrideAttrs (o: {
      failureHook = ''
          ...
        '';
    });
    lispPackages = builtins.mapAttrs (n: v: updateDerivation v) baseLispPackages;
    pkg = builtins.getAttr lispPackage lispPackages;
    in

but I don't think this works because the modified derivations will include references to old unmodified definitions of their dependencies. Somehow I need to have the updated derivations all refer to each other.

I have a few ideas of how to approach this...

1. Add an API function to return Lisp package definitions as attribute sets so that they can be modified before calling build-asdf-system.
2. Make Lisp package definitions into functions that are called with callPackage to resolve their dependencies before creating their derivation.
3. Stealing some clever ideas from other languages' packagings...

But I'm not sure which approach makes sense so I'd be glad for some advice.

[Uthar]

There's no such API now, I don't have any advice , but I will put it on my brain's to-do list :)

There is a similiar thing when overriding dependencies of Quicklisp imported packages with manually specified ones: https://github.com/Uthar/nix-cl/blob/60dbe8f643eced491762bca2cf2ff5b0f998fdb3/packages.nix#L50-L52

[Uthar]

They have an example with Python packages on the wiki https://nixos.wiki/wiki/Overlays#Python_Packages_Overlay

[lukego]

@Uthar I suspect that we need to make some changes to the structure of the packages if we want to do python-style overlays. I wrote some thoughts at https://github.com/NixOS/nixpkgs/pull/193754#issuecomment-1272877277. Do you agree @Uthar or do I misunderstand?

[Uthar]

It sounds good to make it callPackage friendly. It will also be more in-line with other code in nixpkgs, like the pythonPackages. I don't know how to do this though and have no alternative idea for now.

[lukego]

Thanks for the sanity-check. I'll see if I can make a proof-of-concept to understand the topic better. I am still figuring out how the various nixpkgs overriding mechanisms really work.

[Uthar]

hmm maybe changing the api a little bit, from sbclPackages -> sbcl.pkgs will make it easier, then one could just sbcl.overrideAttrs -> packageOverrides like the python example

[Uthar]

maybe a mkLisp function that "adds" pkgs attr set and withPackages function to sbcl

[Uthar]

We should check out lib.makeScope:

Make a set of packages with a common scope. All packages called         
with the provided `callPackage' will be evaluated with the same         
arguments. Any package in the set may depend on any other. The          
`overrideScope'` function allows subsequent modification of the package 
set in a consistent way, i.e. all packages in the set will be           
called with the overridden packages. The package sets may be            
hierarchical: the packages in the set are called with the scope         
provided by `newScope' and the set provides a `newScope' attribute      
which can form the parent scope for later package sets.         

[Uthar]

@lukego Do you want to test your use case again on the scope branch?

Example from nix-cl git repo:

$ nix repl
Welcome to Nix 2.11.1. Type :? for help.

nix-repl> :lf .                      
Added 13 variables.

nix-repl> lib.sbclPackages.bordeaux-threads.lispLibs
[ «derivation /nix/store/sd4zrcywjca98v34r6p71i6lvd1r0i1m-alexandria-20220707-git.drv» ]

nix-repl> lib.sbclPackages.alexa                    
lib.sbclPackages.alexa                        lib.sbclPackages.alexandria_plus
lib.sbclPackages.alexa-tests                  lib.sbclPackages.alexandria_plus_slash_tests
lib.sbclPackages.alexandria                   lib.sbclPackages.alexandria_slash_tests
nix-repl> lib.sbclPackages.alexandria
«derivation /nix/store/sd4zrcywjca98v34r6p71i6lvd1r0i1m-alexandria-20220707-git.drv»

nix-repl> :lf nixpkgs
Added 14 variables.

nix-repl> pkgs = legacyPackages.x86_64-linux

nix-repl> :lf .                                      
Added 13 variables.

nix-repl> new = lib.sbclPackages.overrideScope' (self: super: { alexandria = pkgs.hello; })

nix-repl> new.bordeaux-threads.lispLibs
[ «derivation /nix/store/qcvlk255x98i46cg9vphkdw5pghrizsh-hello-2.12.1.drv» ]

nix-repl> 

[Uthar]

BTW i have documented this new and experimental mechanism in the docs: https://github.com/Uthar/nix-cl/blob/f5290469cc32046617b8debc110efd8650bf8978/doc/api.md

[lukego]

@Uthar How do I pull in this change when I'm working with nixpkgs lispPackages_new?

I'd like to git merge or git cherry-pick but I guess that doesn't work because you are working in a stand-alone repo rather than a fork/branch of nixpkgs, which is what I'm doing?

[Uthar]

yeah both are completely separate

[lukego]

So which one should I use and when? what's the idea?

[Uthar]

I think it would be nice to rm -rf lisp-modules-new and replace with nix-cl, but the api should stabilize a bit first

In nix-cl there's experiments and new stuff and the lisp-modules-new in nixpkgs was ALSO merged as an experiment, so pretty much nothing is stable

[Uthar]

I would just use the nix-cl api because it will probably replace lisp-modules-new

[lukego]

Someday I'd love to read a blog entry or something about the evolution and relationship between ql2nix and lisp-modules and lisp-modules-new and nix-cl... :-) but thanks for the tip, I'll look into migrating onto nix-cl.

[Uthar]

in general I would like something like:

nix build nixpkgs#sbcl.pkgs.alexandria

Just like there are:

nix build nixpkgs#python310.pkgs.pika
nix build nixpkgs#emacs.pkgs.magit

[Uthar]

Of course it's all extremely experimental and can break day by day, until the lisp-modules itself in nixpkgs is replaced with a stable version

[lukego]

Of course

Ahem :-) Here on the outside I have no idea whatsoever what is experimental/working/stable/deprecated/obsolete/etc. Any guidance is greatly appreciated. I have been trying to contribute to lisp-modules-new on upstream nixpkgs and I thought this was The Way Forward and something that you were developing and maintaining.

Now it sounds like nix-cl is a better option for me but at the same time I'd like to get stuff done so an API that's silently changing from day to day doesn't necessarily sound like much fun for me...?

[Uthar]

But That's where the fun is (-:

[Uthar]

but really, i think I'll wait maybe a month and then ask the Lisp guy on nixpkgs to replace it all the way, because now there are overlays, it's fast, overrides etc

[lukego]

There is definitely a lot more of this brand of fun in my life since I started using Nix 😂

[lukego]

Sounds awesome! I'll take nix-cl for a spin soon.

[Uthar]

There is definitely a lot more of this brand of fun in my life since I started using Nix

Yeah, Haskellers eat API breakage for breakfast :-D They're used to it

[lukego]

I haven't done much with Haskell. Smalltalk is my usual benchmark for software being either "alpha" or "obsolete" with almost nothing in between :) "Burn the diskpacks and git repos!"

[lukego]

Question:

The README has instructions for "Build a lisp with packages" but it's based on running inside the nix-cl repo working directory. How would I do this from another repository?

Specifically I'd like to have an external repo with a nix expression that builds a custom Lisp development environment e.g. with all of the Kons-9 dependencies.

Today I'm doing this by referencing <nixpkgs> and using lispPackages_new from there in a shell.nix file: https://github.com/lukego/kons-9/blob/actions/shell.nix but what's the nix-cl way to solve this problem?

[Uthar]

you can do something like this in Nix

let

  nix-cl = builtins.getFlake "github:uthar/nix-cl";

  sbcl = nix-cl.packages.${builtins.currentSystem}.sbcl;

in sbcl.withPackages (ps: [ ps.alexandria ])

Or with flakes, put it in inputs:

{
  description = "test";
  inputs.nix-cl.url = "github:uthar/nix-cl";
  outputs = { self, nixpkgs, nix-cl } : {
  };
}

Or with nix CLI

nix build github:uthar/nix-cl#sbcl.pkgs.alexandria

Oh, the readme example is broken, now I see (sbcl should be pakages.${builtins.currentSystem}.sbcl

[Uthar]

For your case I would use the following flake, and run nix develop

{
  description = "Lisp development environment";

  inputs.nix-cl.url = "github:uthar/nix-cl";

  outputs = { self, nixpkgs, flake-utils, nix-cl }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
        lisps = nix-cl.packages.${system};
        sbcl = lisps.sbcl.withPackages (ps: with ps; [
          closer-mop
          trivial-main-thread
          trivial-backtrace
          cffi
          cl-opengl
          cl-glu
          cl-glfw3
          origin
        ]);
      in {

        devShells.default = pkgs.mkShell {
          buildInputs = [
            sbcl
          ];
        };

      });
}

[Uthar]

You can also pass compile time and run time flags to the lisp:

...
        pkgs = nixpkgs.legacyPackages.${system};
        lisps = nix-cl.packages.${system};
        init = pkgs.writeText "init.lisp" ''
          (pushnew :foo *features*)
          (format t "hello world~%")
        '';
        sbcl = lisps.sbcl.override {
          spec = {
            pkg = lisps.sbcl;
            faslExt = "fasl";
            asdf = lisps.sbcl.pkgs.alexandria.asdf;
            flags = "--dynamic-space-size 8092 --load ${init}";
          };
        };
        sbcl' = sbcl.withPackages (ps: with ps; [
...