Metadata and recurseIntoAttrs

[nagy]

Hi,

eventually, we should add some metadata to the generated packages, like "description" and "license" ( anything else ? ). Currently, this is what I get with the old and the new package set:

nix-repl> lispPackages.cl-async.meta
{ available = true; broken = false; description = "Asynchronous operations for Common Lisp."; insecure = false; name = "lisp-cl-async-20211020-git"; outputsToInstall = [ "out" ]; position = ".../pkgs/development/lisp-modules/define-package.nix:115"; unfree = false; unsupported = false; version = "20211020-git"; }

nix-repl> lispPackages_new.sbclPackages.cl-async.meta
{ available = true; broken = false; insecure = false; name = "cl-async-20211020-git"; outputsToInstall = [ "out" ]; position = ".../pkgs/development/lisp-modules-new/imported.nix:6488"; unfree = false; unsupported = false; }

Metadata is also useful, if we want to be able to make the package set discoverable and eventually make them searchable on https://search.nixos.org.

For that though, we need to call recurseIntoAttrs onto the package set that is bound to the lisp implementation, like sbclPackages. As you can see in you PR https://github.com/NixOS/nixpkgs/pull/172234, ofborg is not seeing anything to rebuild.

[Uthar]

For the metadata we could load-asd it from the Quicklisp source tarballs when dumping the imported.nix file. Then it should be no problem to read back the metadata from imported.nix, unless the source tarball hash has changed.

Or, another idea is to write an importer backend for cl2nix, which already handles such metadata - this would also make us independent from Quicklisp. But Quicklisp comes as a somewhat coherent package, which maybe is desired as opposed to potentially using the tip-of-trunk of each library.

Maybe it's ok to limit this to sbclPackages for now, because there are already ~20% of the Quicklisp packages failing because of missing native libraries or other reasons. Maybe when we fix those we could add CCL, because it's the second most popular one, and later on maybe other implementations.

[teu5us]

I made some progress today with DDO dependencies, which were preventing cl2nix from reading asds. However I found a bunch of projects that load-system something before defining their own systems, which I would also consider DDO dependencies. They are out of definition but still somewhat manageable. What worries me is cffi-grovel in :components (and maybe other packages that allow something similar that I don't know about). It acts fine when used like in osicat, and really bad when used like in cl-libuv

[Uthar]

added recurseIntoAttrs to sbclPackages fb98172376ef1092aa80c07589de0b64738d3f7f

[teu5us]

As far as I can tell, cl2nix will not work the way I thought it would. First, load-asd is not enough to read metadata. You need to find-system afterwards, which will in many cases build defsystem-depends-on stuff and load-system stuff declared in asds as well. Second, to manage 1, one needs to download the entire quicklisp distribution and unpack each release as well as get those systems that are not in quicklisp, even though are required as dependencies, as well as make external dependencies available to the compiler used. I'm working on all of these points, but the code is not available yet

Edit: ASDF drops versions that it's parser doesn't like

[Uthar]

I'm working on all of these points, but the code is not available yet

Good luck and enjoy

I wonder if we'll end up patching asdf

[teu5us]

I already use patched ASDF in my overlay instead of envvars. This made dependency propagation way easier, as it seems to me, since the propagated-build-inputs-like hook had to traverse quite many directories in some cases

[Uthar]

Started some rough work on getting package metadata: branch

[Uthar]

ASDF drops versions that it's parser doesn't like

I encountered the same problem:

WARNING: UIOP/VERSION:PARSE-VERSION: "0.01" contains leading zeros
WARNING: Invalid :version specifier "0.01" for component "kl-verify" from file #P"/nix/store/45w0qgv6nn4iglfafhzcb7pz6w4c365i-kl-verify-20120909-git.tgz/kl-verify.asd", using NIL instead

[Uthar]

Got only around 60% of the data

[Uthar]

Second, to manage 1, one needs to download the entire quicklisp distribution and unpack each release as well as get those systems that are not in quicklisp, even though are required as dependencies, as well as make external dependencies available to the compiler used.

Sounds similar to what lisp-modules in nixpkgs is doing, but this time on a bigger scale

[Uthar]

make external dependencies available to the compiler used

I'll keep working on fixing the remaining quicklisp packages by giving them their native libraries. Then we can get a set of all such libraries needee and make a big nix-shell?

[Uthar]

I even found one system that tries to run GCC in its asd file:

 Error while getting metadata: Error while trying to load
 definition for system cl-fuse from pathname
 /tmp/nix-shell.VLaCA7/cl-fuse.asd:    Subprocess
 #<UIOP/LAUNCH-PROGRAM:PROCESS-INFO {1003384F23}>  with command ("
gcc" "-x" "c" "/tmp/nix-shell.VLaCA7/fuse-launcher.c-minus"                "
-fPIC" "--shared" "-lfuse" "-o"                "
/tmp/nix-shell.VLaCA7/libfuse-launcher.so")  exited with error
 code 

[teu5us]

I'll keep working on fixing the remaining quicklisp packages by giving them their native libraries. Then we can get a set of all such libraries needee and make a big nix-shell?

Do you propagate dependencies?

[teu5us]

Started some rough work on getting package metadata: branch

I wonder how you will handle defsystem-depends-on stuff with asdf:find-system there. Perhaps it would be better to crawl the quickref instead.

[Uthar]

Do you propagate dependencies?

Yes, I borrowed and modified your script: https://github.com/Uthar/nix-cl/blob/master/setup-hook.sh

[Uthar]

I wonder how you will handle defsystem-depends-on stuff with asdf:find-system there.

For now I'm just skipping it

[teu5us]

I also quit using CL_SOURCE_REGISTRY with modified asdf. Plan to use it to replace dependencies in some cases.

Also, for some reason I cannot build lisp packages (in my implementation, it just suddenly stopped working) as I normally would with e.g. python. Had a look at guix and it seems they build directly to $out.