Perl packages for polymake

[mkoeppe]

To simplify installation of polymake, we add some required CPAN packages.

Including recursive dependencies, these are 103 packages, a total of 7.5 MB of source archives, which we pack up as the source tar ball of the new spkg perl_cpan_polymake_prereq. The source tar ball (attached) also contains a copy of the self-contained cpanm Perl script, which is used for installing the packages. If SAGE_CHECK=no, only about 35 packages are installed. The rest of the dependencies are for testing.

Using an spkg-configure.m4, it is downloaded and installed only if the Perl packages are not already present on the system.

The spkg-install installs the packages via the package dependencies mechanism. If some of the packages are already installed in some version on the system, we do not install a newer version.

To try out:

   wget -P upstream https://github.com/sagemath/sage-prod/files/10659280/perl_cpan_polymake_prereq-2019-05-19.tar.gz

See also the sage-devel post about compilation: https://groups.google.com/forum/#!topic/sage-devel/ukJgMweHszU

Depends on #29582

CC: @dimpase @jplab lorenz@math.tu-berlin.de kastner@math.tu-berlin.de paffenholz@mathematik.tu-darmstadt.de

Component: packages: optional

Author: Matthias Koeppe

Branch/Commit: u/mkoeppe/perl_packages_for_polymake @ 225dea9

Issue created by migration from https://trac.sagemath.org/ticket/27763

[dimpase]

comment:2

I'd suggest to have appropriate spkg-configure.m4, to avoid installing extra...

By the way, could you provide spkg-configure.m4 for perl_term_readline_gnu, and other already present as packages polymake deps (perl or not)?

[mkoeppe]

comment:3

Do you have a suggestion how to install these CPAN packages if they are not available? There seems to be a large number of dependencies that each of the listed packages pulls in -- but probably many are "standard" on any system that has perl installed.

[dimpase]

comment:4

I don't know much about perl. Does it have anything analogous to Python's pip, which allows direct installation from the net (from CPAN, I guess) ?

[mkoeppe]

comment:5

Yes, the cpan program does the equivalent of pip.

[dimpase]

comment:6

I tried something like perl -MCPAN -e 'install XML::Writer' but it seems to want to install stuff into /usr/local rather than $SAGE_LOCAL. Do you know how to set all this up properly?

[mkoeppe]

comment:7

I have only used cpan -i, which asks whether to install system wide or in the user's home directory. Here's some info: https://stackoverflow.com/questions/540640/how-can-i-install-a-cpan-module-into-a-local-directory

I used something like this for perl_term_readline_gnu, but this is not going through cpan.

[dimpase]

comment:8

I don't know, it seems one needs to setup some local::lib magic, otherwise I keep getting errors trying to to install into /usr/local.

local::lib is a Perl module that apparently allows one to use non-standard directories for module installation, it might not be in by default (e.g. it's not in Gentoo, one needs to install it).

Anyhow - for checking in autoconf which perl modules are installed, one can use AX_PROG_PERL_MODULES

[dimpase]

comment:9

I also had to install File::Slurp perl module. Fortunately on Gentoo there seems to be an (almost?) 1-to-1 correspondence between perl packages named X::Y and Gentoo packages named X-Y.

I also see

File "src/sage/interfaces/polymake.py", line 884, in sage.interfaces.polymake.Polymake._eval_line
Failed example:
    c                                 # optional - polymake
Expected:
    cube of dimension 15
Got:
    1
    print ref(@SAGE590);
    print reftype(@SAGE590);
**********************************************************************
File "src/sage/interfaces/polymake.py", line 886, in sage.interfaces.polymake.Polymake._eval_line
Failed example:
    c.N_VERTICES                      # optional - polymake
Expected:
    32768
Got:
    Member function 'N_VERTICES' of  object

[mkoeppe]

comment:10

Replying to @dimpase:

I also had to install File::Slurp perl module.

Yes, this is already mentioned in polymake/SPKG.txt

Fortunately on Gentoo there seems to be an (almost?) 1-to-1 correspondence between perl packages named X::Y and Gentoo packages named X-Y.

I also see

File "src/sage/interfaces/polymake.py", line 884, in sage.interfaces.polymake.Polymake._eval_line
Failed example:
    c                                 # optional - polymake
Expected:
    cube of dimension 15
Got:
    1
    print ref(@SAGE590);
    print reftype(@SAGE590);
**********************************************************************
File "src/sage/interfaces/polymake.py", line 886, in sage.interfaces.polymake.Polymake._eval_line
Failed example:
    c.N_VERTICES                      # optional - polymake
Expected:
    32768
Got:
    Member function 'N_VERTICES' of  object

With which version of polymake is this? The 3.4 upgrade ticket has fixes for something like this.

[dimpase]

comment:11

If I do

--- a/src/sage/interfaces/polymake.py
+++ b/src/sage/interfaces/polymake.py
@@ -874,10 +874,6 @@ class Polymake(ExtraTabCompletion, Expect):
         sometimes it hangs, and therefore we remove it from the tests, for now::

             sage: c = polymake.cube(15)             # optional - polymake
-            sage: polymake.eval('print {}->F_VECTOR;'.format(c.name()), timeout=1) # optional - polymake # not tested
-            Traceback (most recent call last):
-            ...
-            RuntimeError: Polymake fails to respond timely

         We verify that after the timeout, polymake is still able to give answers::

then the tests pass, so the timeout thing seems to be doing more harm than good.

[dimpase]

comment:12

I am testing this branch, so 3.4.

[dimpase]

comment:13

oops, sorry, wrong ticket. This is meant to be about polymake update to 3.4

[mkoeppe]

comment:14

yes, #24905.

[dimpase]

comment:15

Please see #27795 for using system's perm package Term::ReadLine::Gnu

[mkoeppe]

Dependencies: #24905, #27795

[dimpase]

comment:17

How about adding checks for all the needed perl packages in the style of #27795 to polymake itself? Actually, it would be easier (per package), as Term::ReadLine::Gnu is non-standard in the sense one cannot just use it.

As usual, they pay a usability price by not using autotools :-)

[mkoeppe]

comment:18

Replying to @dimpase:

How about adding checks for all the needed perl packages in the style of #27795 to polymake itself?

polymake itself does check for the packages needed at build time in its configure script. The MongoDB package is checked at run time of polymake for the optional polyDB feature.

[mkoeppe]

comment:19

The challenge in this ticket is to make the installation of missing Perl packages not depend on internet access. Here's a possible approach.

First install App::cpanminus. Then

cpanm --local-lib-contained $PERL_LOCAL_LIB_ROOT --exclude-vendor --save-dists $PARTIAL_CPAN_MIRROR XML::Writer XML::LibXML XML::LibXSLT File::Slurp Term::ReadLine::Gnu JSON SVG MongoDB

will (1) download and install these prerequisites in $PERL_LOCAL_LIB_ROOT, (2) save the source packages in a directory tree $PARTIAL_CPAN_MIRROR.

These are 58 packages, a total of 5 MB of source archives, which we can pack up as the source tar ball of the new spkg perl_cpan_polymake_prereq.

Then figure out how to install with plain cpan (not cpanm, which is not available out of the box on macOS) using this local partial CPAN mirror.

[mkoeppe]

Branch: u/mkoeppe/perl_packages_for_polymake

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

31b5223

Add perl_cpan_polymake_prereq as a polymake dependency

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Commit: 31b5223

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

476c148	Install CPAN modules with cpanm
7611f43	Use cpanm --skip-satisfied
749c447	List dependencies in Makefile.PL

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 31b5223 to 749c447

[mkoeppe]

Changed dependencies from #24905, #27795 to none

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,2 +1,9 @@
 To simplify installation of polymake, we add some required CPAN packages. 

+Including recursive dependencies, these are 58 packages, a total of 5 MB of source archives, which we pack up as the source tar ball of the new spkg `perl_cpan_polymake_prereq`.
+The source tar ball also contains a copy of the self-contained `cpanm` Perl script, which is used for installing the packages.
+
+Using an `spkg-configure.m4`, it is downloaded and installed only if the Perl packages are not already present on the system.
+
+The `spkg-install` installs the packages via the package dependencies mechanism. If some of the packages are already installed in some version on the system, we do not install a newer version.
+

[mkoeppe]

Author: Matthias Koeppe

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

aecb94a

perl_cpan_polymake_prereq/spkg-src: Actually compress the tarball

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 749c447 to aecb94a

[mkoeppe]

comment:25

Attachment: perl_cpan_polymake_prereq-2019-05-13.tar.gz

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,7 +1,7 @@
 To simplify installation of polymake, we add some required CPAN packages. 

 Including recursive dependencies, these are 58 packages, a total of 5 MB of source archives, which we pack up as the source tar ball of the new spkg `perl_cpan_polymake_prereq`.
-The source tar ball also contains a copy of the self-contained `cpanm` Perl script, which is used for installing the packages.
+The source tar ball (attached) also contains a copy of the self-contained `cpanm` Perl script, which is used for installing the packages.

 Using an `spkg-configure.m4`, it is downloaded and installed only if the Perl packages are not already present on the system.

[jplab]

Attachment: polymake-3.4.log

Curious install log -> perl_term_readline_gnu

[jplab]

comment:27

After I did install successfully polymake, the interface was broken (compile error of some sort). And tab completion did not work. Then I saw that perm_term_readline_gnu experimental package was not installed (although it seems like the configure step of the installation detected the local package but seemed like it can not use the local package).

Does that make sense? Unfortunately, I can not offer more traceback, I reached the end of my terminal buffer.

If needed, I could try to proceed similarly on a twin computer to see if I can reproduce this.

[dimpase]

comment:28

By local package you mean system package, right?

Can you uninstall Sage's perm_term_readline_gnu and then in terminal run

perl -e "use Term::ReadLine; Term::ReadLine->get_all_function_names" 

• there should be no output if perm_term_readline_gnu is installed, and an error message otherwise.

[dimpase]

comment:29

Also, run ./configure and see whether you have

    perl_term_readline_gnu-1.35 will not be installed (configure check)

in the output

[mkoeppe]

comment:30

I think it's possible that the system's Term::ReadLine::Gnu becomes unusable in the sage environment if sage installs its own libreadline. If for some reason we install our own libreadline, then we should probably also install Term::ReadLine::Gnu.

[mkoeppe]

comment:31

Or we could just revert #27795.

[dimpase]

comment:32

when I wrote #27795 I was under impression that perl is isolated from sage's "vendored madness" libraries, as it is an independent executable, just like, say, bash.

if the problem is indeed as you described then the right way forward is to get #27277 reviewed and merged.then readline wont get built if it is available.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

5916141	Fix polymake lrslib detection
857af80	polymake: add dependency ppl
10189db	Merge branch 't/27803/public/27803-polymake-lrslib-detection' into t/27763/perl_packages_for_polymake

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from aecb94a to 10189db

[mkoeppe]

comment:34

Adding #27803 dependency to prevent merge conflict.

[mkoeppe]

Dependencies: #27803

[mkoeppe]

comment:35

Replying to @dimpase:

when I wrote #27795 I was under impression that perl is isolated from sage's "vendored madness" libraries, as it is an independent executable, just like, say, bash.

Yes, the Perl executable is fine, but the myriad of CPAN packages link to lots of libraries.

if the problem is indeed as you described then the right way forward is to get #27277 reviewed and merged.then readline wont get built if it is available.

OK.

[dimpase]

comment:36

Replying to @mkoeppe:

Replying to @dimpase:

when I wrote #27795 I was under impression that perl is isolated from sage's "vendored madness" libraries, as it is an independent executable, just like, say, bash.

Yes, the Perl executable is fine, but the myriad of CPAN packages link to lots of libraries.

yes, I understand this, but I imagine the only conflicts of this sort are possible if you run polymake as a python extension, that loads a wrong dll at runtime.

As I don't know how the polymake interface works, I can't tell whether this is the case.

In any case, could you explain where these *.so parts of perl packages may be found, so that I can look at them with ldd/otool ?

[dimpase]

comment:37

If there is indeed a possible problem like this, build/pkgs/perl_term_readline_gnu/spkg-configure.m4 needs to be modified to include a check for readline.

I would actually be more worried about a potential conflict between an incompatible readline library, say, on OSX, and the one built by Sage (or provided by Homebrew/MacPorts)...

[jplab]

comment:38

Replying to @dimpase:

Also, run ./configure and see whether you have

    perl_term_readline_gnu-1.35 will not be installed (configure check)

in the output

Yes, I met system package.

Ok, so here is the outcome.

I ran the command

perl -e "use Term::ReadLine; Term::ReadLine->get_all_function_names"

before uninstalling perl_term_readline_gnu: No output.

After uninstalling (inside of sage shell using the sage-spkg-uninstall command) I still get no output.

Things now seems to work flawlessly using the interface though.

Running ./configure, I do check the (configure check).

Ok, so now it seems to catch the system Term ReadLine. I believe I know how it happened: I did install the perl libraries using cpan -i XML::Writer XML::LibXML XML::LibXSLT File::Slurp JSON SVG as the sage -info polymake suggested, but then, the installation of the polymake package was not preceeded by a rerun of ./configure to catch the libraries that are now meanwhile to be found in my PATH? Somehow Sage still relied on its own perl_term_readline_gnu although the system one was found so it did not install the dependency. Weird.

If I run it on a clean computer, without the spkg perl_term_readline_gnu (and without any installation of the required perl libraries) I do get the error message (that now looks quite familiar to the one I was getting in Sage originally).

Ok, so in the end, I do not think there is an issue: I'll test it on fresh untouched debian machines to have a clean feedback.

[dimpase]

comment:39

Isn't the presence of local::lib module a pre-req for all these package installations to function?

[dimpase]

comment:40

Replying to @jplab:

Replying to @dimpase:

Also, run ./configure and see whether you have

    perl_term_readline_gnu-1.35 will not be installed (configure check)

in the output

Yes, I met system package.

Ok, so here is the outcome.

I ran the command

perl -e "use Term::ReadLine; Term::ReadLine->get_all_function_names"

before uninstalling perl_term_readline_gnu: No output.

After uninstalling (inside of sage shell using the sage-spkg-uninstall command) I still get no output.

Things now seems to work flawlessly using the interface though.

Running ./configure, I do check the (configure check).

Ok, so now it seems to catch the system Term ReadLine. I believe I know how it happened: I did install the perl libraries using cpan -i XML::Writer XML::LibXML XML::LibXSLT File::Slurp JSON SVG as the sage -info polymake suggested, but then, the installation of the polymake package was not preceeded by a rerun of ./configure to catch the libraries that are now meanwhile to be found in my PATH? Somehow Sage still relied on its own perl_term_readline_gnu although the system one was found so it did not install the dependency. Weird.

well, this is by (somewhat unfortunate) design. Once you updated ./configure by installing/updating a spkg-configure.m4 file in build/pkgs/foo/ and you have foo installed in Sage and as a system package, the only way to switch from Sage-installed to system foo is to manually uninstall Sage's foo, and then re-run configure (othewise configure would check, and succeed in this, for Sage's foo first).

The main reason for that that some people do not want to retire the old sage -f foo as a way to install/update a package, see e.g. #27373...

[jplab]

comment:41

well, this is by (somewhat unfortunate) design. Once you updated ./configure by installing/updating a spkg-configure.m4 file in build/pkgs/foo/ and you have foo installed in Sage and as a system package, the only way to switch from Sage-installed to system foo is to manually uninstall Sage's foo, and then re-run configure (othewise configure would check, and succeed in this, for Sage's foo first).

The main reason for that that some people do not want to retire the old sage -f foo as a way to install/update a package, see e.g. #27373...

Oh I see now. Got it! Good to know.

[mkoeppe]

comment:42

Replying to @dimpase:

Isn't the presence of local::lib module a pre-req for all these package installations to function?

For installing the modules, yes. But the cpanm script is fatpacked with all its prerequisite libraries, including local::lib.

For using the modules, no. We just modify the environment variable PERL5LIB.

[dimpase]

comment:43

surely, local::lib is an installation tool, but can you install it into system wide perl without root access?