Prefer Linux package manager over linuxbrew; add nix detection

[tobiasdiez]

If brew is installed on Linux, Sagemath thinks this is the primary package manager and, e.g., configure suggests to run brew ... to install new packages. However, linuxbrew is untested as a platform for Sage, and so this is not a good idea.

We change build/bin/sage-guess-package-system so that brew is only tested after the Linux package managers.

We also add a detection for nix-env.

CC: @mkoeppe @collares @jplab

Component: build: configure

Author: Matthias Koeppe

Branch/Commit: u/mkoeppe/prefer_linux_package_manager_over_linuxbrew @ 9af33a1

Issue created by migration from https://trac.sagemath.org/ticket/32753

[dimpase]

comment:2

I now see this in our gitpod.

[mkoeppe]

Branch: u/mkoeppe/prefer_linux_package_manager_over_linuxbrew

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

Commit: 90f7ced

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,3 +1,7 @@
-If brew is installed on Linux, Sagemath thinks this is the primary package manager and, e.g., configure suggests to run brew xyz to install new packages.
+If brew is installed on Linux, Sagemath thinks this is the primary package manager and, e.g., `configure` suggests to run `brew ...` to install new packages. However, linuxbrew is untested as a platform for Sage, and so this is not a good idea.

-Probably build/bin/sage-guess-package-system should be changed so that brew is only tested after the Linux package managers. 
+We change `build/bin/sage-guess-package-system` so that `brew` is only tested after the Linux package managers. 
+
+We also add a detection for `nix-env`.
+
+

[mkoeppe]

comment:4

For nix, do we need to check whether a nix environment is active, like we do for conda?

---

New commits:

04f9749	build/bin/sage-guess-package-system: check linuxbrew after linux package managers because it is untested
90f7ced	build/bin/sage-guess-package-system: Check for nix

[collares]

comment:5

To be honest with you, I have never used nix-env (the manual offers a declarative path and a nix-env path, and strongly suggests nix-env is the worst of the two alternatives). I think supporting nix-env is valid, but given my lack of experience with it I am not sure what the workflow would be in this case. You can certainly install packages with nix-env in your "default profile", and the executables would be in PATH without having to activate an environment.

People who follow the declarative approach and want to build Sage from source would bypass nix-env and write a derivation (a declarative recipe, basically) to pull in the required dependencies, and then pass this to nix-shell to obtain an ephemeral virtualenv-like environment with those deps available. Or, instead of writing the derivation themselves, they could also use Nixpkgs's Sage derivation to get the same build environment we use. Either way, they would be able to build Sage from Git sources using Nix packages for the dependencies, but nix-env would be none the wiser.

It would be useful to know a few things so I can understand how this fits into the Nix model: Is it just for printing a command to install dependencies? Or does it query installed packages as well? What happens if Singular (say) just exists on PATH but nix-env doesn't report it as installed? Pointing me to the relevant source file would be enough, no need to answer in detail. Thanks!

[mkoeppe]

comment:6

Replying to @collares:

People who follow the declarative approach and want to build Sage from source would bypass nix-env and write a derivation (a declarative recipe, basically) to pull in the required dependencies, and then pass this to nix-shell to obtain an ephemeral virtualenv-like environment with those deps available. Or, instead of writing the derivation themselves, they could also use Nixpkgs's Sage derivation to get the same build environment we use.

It would be great to have instructions how to do Sage development using nixpkgs in our installation manual.

Other than figuring out how to provision a nix environment with packages on top of the nixos/nix Docker image (so I can say tox -e docker-nixos-standard), I have no experience with nix.

By the way, I am getting the error:

Step 6/46 : RUN  nix-env --install  gd iml m4ri gfan singular mpfi cmake symmetrica flintqs freetype cddlib libbraiding brial ppl sympow zn_poly readline R sqlite tachyon giac m4rie blas lapack ntl glpk fflas-ffpack libmpc gsl boost nauty binutils gnumake gnum4 perl python3 gnutar bc gcc bash flint arb pari cliquer libhomfly openssl gengetopt planarity lrcalc palp ecm givaro ecl eclib lcalc rankwidth qhull gettext autoconf automake libtool pkg-config
 ---> Running in 0dcbda1f2003
error: selector 'libhomfly' matches no derivations

so it may be that some work on the nix.txt files is needed.

[mkoeppe]

comment:7

Replying to @collares:

It would be useful to know a few things so I can understand how this fits into the Nix model: Is it just for printing a command to install dependencies?

Yes, it's for the system package advice that is printed at the end of a configure run.

The code for this is in build/bin/sage-print-system-package-command.

[mkoeppe]

comment:8

Replying to @collares:

What happens if Singular (say) just exists on PATH

configure only checks what is available in PATH.

For packages where we don't find a system package in this way, if there is system package information in build/pkgs/*/nix.txt for the package, we issue the system package advice. No package manager queries are done -- this is all left to the user.

(This is a design principle for this code -- it does not attempt to take care of system package installation tasks for the user; instead it only informs/educates the user about possible commands.)

[collares]

comment:9

Thanks for the information, this is very helpful! So, yeah, I think recommending nix-env is fine, since people who use the declarative approach can just copy and paste the list of packages into their derivations.

As for the libhomfly error: The preferred way to install packages using nix-env is nix-env -f '<nixpkgs>' -iA m4ri libhomfly (say). The -A flag specifies a saner way of finding the relevant package; in this case, the -A-less version caught a typo in the Nix package. I've submitted a PR to fix it in Nixpkgs, but ideally the nix-env invocation would be switched to something like the command I mentioned above.

I will try to write some docs for Sage development with nixpkgs as part of the Sage 9.6 release.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

e288628

build/bin/sage-print-system-package-command (nix): Use nix-env --install --attr

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 90f7ced to e288628

[mkoeppe]

comment:11

Not sure what the -f '<nixpkgs>' is - is there something I need to add?

[mkoeppe]

comment:12

Replying to @collares:

The -A flag specifies a saner way of finding the relevant package

Thanks, this helped!

[mkoeppe]

comment:13

configure now gives the following advice:

checking for the package system in use... nix
configure:

    hint: installing the following system packages, if not
    already present, is recommended and may avoid having to
    build them (though some may have to be built anyway):

      $ nix-env --install --attr ecl eclib fflas-ffpack giac givaro gsl iml lcalc libhomfly blas lapack palp pari R

configure:

    hint: installing the following system packages, if not
    already present, may provide additional optional features:

      $ nix-env --install --attr ffmpeg graphviz imagemagick texinfo graphviz libxml2 lrs pdf2svg polymake

[collares]

comment:14

This is perfect, many thanks! I don't know why I needed the -f <nixpkgs> here, perhaps because I am on NixOS? It's best to leave it out if it works outside NixOS.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

3d96dc5

build/bin/sage-print-system-package-command, build/bin/write-dockerfile.sh: Use nix-env -f '' --install --attr

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from e288628 to 3d96dc5

[mkoeppe]

comment:16

OK, looks like I do need this switch

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 3d96dc5 to b07fece

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

b07fece

build/pkgs/gfortran/distros/nix.txt: New

[mkoeppe]

comment:18

Now I am getting

building '/nix/store/r2w1z0912rilh4l1yyg9jv3d66bnjk99-user-environment.drv'...
error: packages '/nix/store/9iry5bm9cxdbspap6qi835g67kvdninx-boost-1.77.0/lib/libboost_thread.so' and '/nix/store/6x1m911w8378qbxjf9pcli9hijrqx1km-nix-2.6.0/lib/libboost_thread.so' have the same priority 5; use 'nix-env --set-flag priority NUMBER INSTALLED_PKGNAME' to change the priority of one of the conflicting packages (0 being the highest priority)
error: builder for '/nix/store/r2w1z0912rilh4l1yyg9jv3d66bnjk99-user-environment.drv' failed with exit code 1
The command '/bin/sh -c nix-env -f '<nixpkgs>' --install --attr  gd iml m4ri gfan gfortran singular mpfi cmake symmetrica flintqs freetype cddlib libbraiding brial ppl sympow zn_poly readline R sqlite tachyon giac m4rie blas lapack ntl glpk fflas-ffpack libmpc gsl boost nauty binutils gnumake gnum4 perl python3 gnutar bc gcc bash flint arb pari cliquer libhomfly openssl gengetopt planarity lrcalc palp ecm givaro ecl eclib lcalc libatomic_ops rankwidth qhull gettext autoconf automake libtool pkg-config' returned a non-zero code: 100

[collares]

comment:19

I've never seen this error and I am not exactly sure how to fix it. Can we just remove Boost's nix.txt, as a workaround? Hopefully there are no other conflicts like this.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

9af33a1

build/pkgs/boost_cropped/distros/nix.txt: Remove

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from b07fece to 9af33a1

[mkoeppe]

comment:21

That worked, thanks. Next error:

building '/nix/store/4vmcjc3jqzvzsy1mvydsl6k5glbbvvvm-user-environment.drv'...
error: packages '/nix/store/cm5d8zx8jf24f6vjxzdx9by3k6rn5zd2-gfortran-wrapper-9.3.0-man/share/man/man7/fsf-funding.7.gz' and '/nix/store/310zm0mxwmk7ynsk0qjb7hiaf734s8p8-gcc-wrapper-10.3.0-man/share/man/man7/fsf-funding.7.gz' have the same priority 10; use 'nix-env --set-flag priority NUMBER INSTALLED_PKGNAME' to change the priority of one of the conflicting packages (0 being the highest priority)
error: builder for '/nix/store/4vmcjc3jqzvzsy1mvydsl6k5glbbvvvm-user-environment.drv' failed with exit code 1
The command '/bin/sh -c nix-env -f '<nixpkgs>' --install --attr  gd iml m4ri gfan gfortran singular mpfi cmake symmetrica flintqs freetype cddlib libbraiding brial ppl sympow zn_poly readline R sqlite tachyon giac m4rie blas lapack ntl glpk fflas-ffpack libmpc gsl nauty binutils gnumake gnum4 perl python3 gnutar bc gcc bash flint arb pari cliquer libhomfly openssl gengetopt planarity lrcalc palp ecm givaro ecl eclib lcalc libatomic_ops rankwidth qhull gettext autoconf automake libtool pkg-config' returned a non-zero code: 100

[mkoeppe]

comment:24

Perhaps good enough for merging?

[mkoeppe]

Branch has merge conflicts