Add/update the system package information for the "_recommended" dummy package

[mkoeppe]

Follow-up from #29557 (Add script package _recommended and generate "recommended system packages" info).

From egourgoulhon #30624 and similar reports by gh-tobiasdiez, jhpalmieri #29557:

configure ends with the following recommendations:

  configure: hint: installing the following system packages is recommended and may 
avoid building some of the above SPKGs from source:
configure:   $ sudo apt-get update 
  $ sudo apt-get install texlive-generic-extra texlive-xetex latexmk pandoc dvipng 
default-jdk ffmpeg libavdevice-dev libcdd-dev libcdd-tools libec-dev eclib-tools 
libgc-dev libgiac-dev xcas pari-gp2c lcalc liblfunction-dev libnauty-dev nauty 
pari-gp2c libpari-dev pari-doc pari-elldata pari-galdata pari-galpol pari-seadata 
sympow

• 30910 replaced texlive-generic-extra by a package that exists in modern Debian distributions.

• 31042 addresses pandoc.

• 29372 changed the message, showing installation hints for non-enabled optional packages separately.

• 32956 create dummy script packages for ffmpeg and imagemagick

The message is printed regardless of whether ffmpeg et al. are installed because build/pkgs/_recommended/spkg-configure.m4 does not check anything.

To this end, tests need to be added for the presence of the various recommended packages in the system.

We do this by splitting out individual script packages, similar to the existing build/pkgs/pandoc and build/pkgs/graphviz.

Depends on #30910 Depends on #31042

CC: @jhpalmieri @egourgoulhon @tobiasdiez @seblabbe @slel

Component: build: configure

Keywords: sd111

Issue created by migration from https://trac.sagemath.org/ticket/30930

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,3 +1,22 @@
 Follow-up from #29557 (Add script package _recommended and generate "recommended system packages" info).

+From egourgoulhon #30624:
+
+  FWIW, I've installed Sage 9.3.beta0 from scratch on Ubuntu 20.04; 
+  configure ends with the following recommendations:
+
+```
+  configure: hint: installing the following system packages is recommended and may 
+avoid building some of the above SPKGs from source:
+configure:   $ sudo apt-get update 
+  $ sudo apt-get install texlive-generic-extra texlive-xetex latexmk pandoc dvipng 
+default-jdk ffmpeg libavdevice-dev libcdd-dev libcdd-tools libec-dev eclib-tools 
+libgc-dev libgiac-dev xcas pari-gp2c lcalc liblfunction-dev libnauty-dev nauty 
+pari-gp2c libpari-dev pari-doc pari-elldata pari-galdata pari-galpol pari-seadata 
+sympow
+```
+  Now, as mentioned in ​[#29557 comment:56](https://github.com/sagemath/sage/issues/29557#comment:56), texlive-generic-extra does not exist on Ubuntu 20.04 (there is texlive-latex-extra, which is already installed on my system). Moreover the recommended packages texlive-xetex, latexmk, pandoc, dvipng, default-jdk, ffmpeg, libavdevice-dev, libcdd-dev and libcdd-tools are already installed on my system.
+
+From gh-tobiasdiez #29557:
+  The package texlive-generic-extra doesn't exist for ubuntu 20.04, and on 18.04 it was already labeled as "transitional dummy package". I'm not sure what's the correct replacement, maybe simply "texlive"?

[mkoeppe]

Description changed:

--- 
+++ 
@@ -20,3 +20,19 @@

 From gh-tobiasdiez #29557:
   The package texlive-generic-extra doesn't exist for ubuntu 20.04, and on 18.04 it was already labeled as "transitional dummy package". I'm not sure what's the correct replacement, maybe simply "texlive"?
+
+From jhpalmieri #29557:
+  On OS X, I see this:
+
+```
+configure:40015: notice: the following SPKGs did not find equivalent system packages: _recommended arb brial cbc cddlib cliquer cmake coxeter3 eclib ecm fflas_ffpack flint flintqs fplll gf2x gfan giac givaro gp2c iml lcalc libbraiding libsemigroups lrcalc m4ri m4rie ntl palp pari pari_elldata pari_galdata pari_galpol pari_nftables pari_seadata pari_seadata_small perl_cpan_polymake_prereq perl_term_readline_gnu planarity rw symmetrica sympow tachyon zn_poly
+configure:40017: checking for the package system in use
+configure:40020: result: homebrew
+configure:40029: hint: installing the following system packages is recommended and may avoid building some of the above SPKGs from source:
+configure:40031:   $ brew install pandoc ffmpeg imagemagick texinfo cmake
+```
+  On a follow-up, we could improve this message. First, these recommended packages don't actually help to avoid building any of the listed SPKGs. Second, this message is printed regardless of whether ffmpeg et al. are installed. Third, I actually have pandoc installed, it is correctly detected by ./configure, but it recommends installing it anyway.
+  Replying to mkoeppe:
+    Note there's also a separate pandoc script package
+  Right, and that correctly detects my pandoc installation, so it would be nice if the ending message took that into account. The ending message automatically prints every recommended external package, even those which are installed and have corresponding script packages.
+

[mkoeppe]

comment:3

The debian package providing latex is updated in #30910, which I have made a dependency.

[mkoeppe]

Dependencies: #30910

[mkoeppe]

Branch: u/mkoeppe/add_update_the_system_package_information_for_the_recommendeddummy_package

[mkoeppe]

Commit: afdd20f

[mkoeppe]

comment:5

This is taking care of the issue with pandoc.

---

Last 10 new commits:

050dcb8	tox.ini (local-sudo): Use --yes --no-install-recommends
c8fbe0b	tox.ini (local-sudo): Ignore errors when IGNORE_MISSING_SYSTEM_PACKAGES=yes
ff34897	tox.ini (local): Guess the package system if it is not provided as a factor
e27ff19	Merge branch 'u/mkoeppe/tox_ini__add_environments_local_sudo_ubuntu_standard__etc_' of git://trac.sagemath.org/sage into public/build/fixwsl
6ff45c4	Cleanup workflow
5348144	Replace texlive-generic-extra
0fd029f	Narrow down events that trigger the workflow
3ccae02	build/pkgs/_recommended/distros/debian.txt: Update latex package
6d0e673	Merge branch 't/30910/public/build/fixwsl' into t/30930/add_update_the_system_package_information_for_the___recommended__dummy_package
afdd20f	build/pkgs/_recommended/distros, src/doc/bootstrap: Remove duplication of pandoc

[mkoeppe]

comment:6

That configure keeps suggesting packages from the _recommended dummy package is because build/pkgs/_recommended/spkg-configure.m4 does not check anything.

To improve this, tests need to be added for the presence of the various recommended packages in the system. Either in build/pkgs/_recommended/spkg-configure.m4 or in individual "script packages" similar to build/pkgs/pandoc.

[mkoeppe]

Author: Matthias Koeppe

[dimpase]

comment:8

OK, this does the trick for pandoc. I guess ffmpeg, imagemagick, texinfo need semi-dummy packages created (i.e. with spkg-configure checking their presence, but without real spkg-install), right?

[mkoeppe]

comment:9

Replying to @dimpase:

OK, this does the trick for pandoc. I guess ffmpeg, imagemagick, texinfo need semi-dummy packages created (i.e. with spkg-configure checking their presence, but without real spkg-install), right?

That's right. Likewise for libxml in #30899.

[dimpase]

comment:10

Replying to @mkoeppe:

Replying to @dimpase:

OK, this does the trick for pandoc. I guess ffmpeg, imagemagick, texinfo need semi-dummy packages created (i.e. with spkg-configure checking their presence, but without real spkg-install), right?

That's right. Likewise for libxml in #30899.

should this be done on this ticket, or rather elsewhere?

[mkoeppe]

comment:11

Fine to do it here if it's not holding up this ticket too long

[mkoeppe]

comment:13

Replying to @mkoeppe:

Likewise for libxml in #30899.

This is now done in #27921

[mkoeppe]

comment:14

Needs review

[mkoeppe]

Changed keywords from none to sd111

[mkoeppe]

comment:15

Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111

[seblabbe]

comment:16

The branch has a conflict on top of 9.3.beta3 (I confirm there is one)

[seblabbe]

Work Issues: branch has conflict

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from afdd20f to bdd5b9d

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

bdd5b9d

Merge tag '9.3.beta3' into t/30930/add_update_the_system_package_information_for_the___recommended__dummy_package

[mkoeppe]

Changed work issues from branch has conflict to none

[seblabbe]

comment:20

I tested the branch. make build works.

With the branch, I see the following message on my computer (Ubuntu 20.04):

    notice: the following SPKGs did not find equivalent system packages:

        _recommended boost coxeter3 gp2c igraph isl libsemigroups pari_elldata pari_galpol pari_nftables pari_seadata

checking for the package system in use... debian
configure:

    hint: installing the following system packages, if not
    already present, is recommended and may avoid having to
    build them (though some may have to be built anyway):

      $ sudo apt-get update 
      $ sudo apt-get install  texlive-latex-extra texlive-xetex latexmk dvipng default-jdk ffmpeg libavdevice-dev libboost-dev pari-gp2c libigraph-dev libisl-dev

    After installation, re-run configure using:

      $ ./config.status --recheck && ./config.status

And tox -e docker-archlinux-minimal -- config.status gives this:

    notice: the following SPKGs did not find equivalent system packages:

        _recommended arb boost boost_cropped brial cbc cddlib cliquer cmake coxeter3 eclib ecm fflas_ffpack flint flintqs fplll freetype gc gf2x gfan gfortran giac git givaro glpk gp2c gsl igraph iml isl lcalc libatomic_ops libbraiding libgd libhomfly libnauty libpng libsemigroups lrcalc m4ri m4rie mpfi nauty ninja_build ntl openblas palp pandoc pari pari_elldata pari_galdata pari_galpol pari_nftables pari_seadata pari_seadata_small patch perl_cpan_polymake_prereq perl_term_readline_gnu planarity ppl python3 r rw suitesparse symmetrica sympow tachyon tox zeromq zn_poly

checking for the package system in use... arch
configure:

    hint: installing the following system packages, if not
    already present, is recommended and may avoid having to
    build them (though some may have to be built anyway):

      $ sudo pacman -Sy
      $ sudo pacman -S  arb boost boost brial coin-or-cbc cddlib coxeter eclib fflas-ffpack flintqs gc gf2x gfan gcc-fortran libgiac giac glpk gsl igraph iml lcalc libatomic_ops libbraiding gd libhomfly lrcalc m4ri m4rie nauty ninja openblas lapack cblas palp pandoc pari pari-galdata pari-seadata pari-elldata pari-galpol pari-elldata pari-galdata pari-galpol pari-seadata pari-seadata patch perl-term-readline-gnu planarity ppl r rankwidth suitesparse symmetrica sympow tachyon zn_poly

    After installation, re-run configure using:

      $ ./config.status --recheck && ./config.status

We see that pandoc still appears above. Is this what we want?

Could we update the description of the ticket (which is too large), because I don't know what the branch is suppose to do exactly?

[mkoeppe]

Changed commit from bdd5b9d to none

[mkoeppe]

Changed branch from u/mkoeppe/add_update_the_system_package_information_for_the_recommendeddummy_package to none

[mkoeppe]

comment:21

Replying to @seblabbe:

Could we update the description of the ticket (which is too large), because I don't know what the branch is suppose to do exactly?

I have instead split out the pandoc part (which is the only thing that the current branch took care of) out into the separate ticket #31042.

[mkoeppe]

Changed dependencies from #30910 to #30910, #31042

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,9 +1,8 @@
 Follow-up from #29557 (Add script package _recommended and generate "recommended system packages" info).

-From egourgoulhon #30624:
+From egourgoulhon #30624 and similar reports by gh-tobiasdiez, jhpalmieri #29557:

-  FWIW, I've installed Sage 9.3.beta0 from scratch on Ubuntu 20.04; 
   configure ends with the following recommendations:

@@ -16,23 +15,19 @@ pari-gp2c libpari-dev pari-doc pari-elldata pari-galdata pari-galpol pari-seadata sympow

-  Now, as mentioned in ​[#29557 comment:56](https://github.com/sagemath/sage/issues/29557#comment:56), texlive-generic-extra does not exist on Ubuntu 20.04 (there is texlive-latex-extra, which is already installed on my system). Moreover the recommended packages texlive-xetex, latexmk, pandoc, dvipng, default-jdk, ffmpeg, libavdevice-dev, libcdd-dev and libcdd-tools are already installed on my system.

-From gh-tobiasdiez #29557:
-  The package texlive-generic-extra doesn't exist for ubuntu 20.04, and on 18.04 it was already labeled as "transitional dummy package". I'm not sure what's the correct replacement, maybe simply "texlive"?
+- #30910 replaced `texlive-generic-extra` by a package that exists in modern Debian distributions.
+- #31042 addresses `pandoc`.

-From jhpalmieri #29557:
-  On OS X, I see this:
+On this ticket, we improve the message further.

-```
-configure:40015: notice: the following SPKGs did not find equivalent system packages: _recommended arb brial cbc cddlib cliquer cmake coxeter3 eclib ecm fflas_ffpack flint flintqs fplll gf2x gfan giac givaro gp2c iml lcalc libbraiding libsemigroups lrcalc m4ri m4rie ntl palp pari pari_elldata pari_galdata pari_galpol pari_nftables pari_seadata pari_seadata_small perl_cpan_polymake_prereq perl_term_readline_gnu planarity rw symmetrica sympow tachyon zn_poly
-configure:40017: checking for the package system in use
-configure:40020: result: homebrew
-configure:40029: hint: installing the following system packages is recommended and may avoid building some of the above SPKGs from source:
-configure:40031:   $ brew install pandoc ffmpeg imagemagick texinfo cmake
-```
-  On a follow-up, we could improve this message. First, these recommended packages don't actually help to avoid building any of the listed SPKGs. Second, this message is printed regardless of whether ffmpeg et al. are installed. Third, I actually have pandoc installed, it is correctly detected by ./configure, but it recommends installing it anyway.
-  Replying to mkoeppe:
-    Note there's also a separate pandoc script package
-  Right, and that correctly detects my pandoc installation, so it would be nice if the ending message took that into account. The ending message automatically prints every recommended external package, even those which are installed and have corresponding script packages.
+- First, these recommended packages don't actually help to avoid building any of the listed SPKGs. 

+  So it could be argued that `./configure` should recommend them separately. This is similar to #29372 "At the end of `configure`, show installation hints for disabled optional packages separately".
+
+
+- Second, this message is printed regardless of whether `ffmpeg` et al. are installed because build/pkgs/_recommended/spkg-configure.m4 does not check anything.
+
+  To this end, tests need to be added for the presence of the various recommended packages in the system. Either in `build/pkgs/_recommended/spkg-configure.m4` or in individual script packages similar to `build/pkgs/pandoc`.
+
+

[mkoeppe]

Changed author from Matthias Koeppe to none

[mkoeppe]

comment:23

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

[slel]

comment:24

Should _recommended really be listed in this configure notice?

    notice: the following SPKGs did not find equivalent system packages:

        _recommended cbc coxeter3 gp2c ...

[mkoeppe]

comment:25

Yes, it may look funny but at least one can see that way where the system package recommendations are coming from

[slel]

comment:26

Makes sense.

[dimpase]

comment:27

texinfo keeps popping up on Homebrew for some reason --- perhaps, because texinfo is not a program, there no such executable anymore, it's replaced by tex2any - but the package name remains texinfo.

[mkoeppe]

comment:28

From the ticket description: To this end, tests need to be added for the presence of the various recommended packages in the system. Either in build/pkgs/_recommended/spkg-configure.m4 or in individual script packages similar to build/pkgs/pandoc.

[mkoeppe]

Description changed:

--- 
+++ 
@@ -18,16 +18,14 @@

 - #30910 replaced `texlive-generic-extra` by a package that exists in modern Debian distributions.
 - #31042 addresses `pandoc`.
-
-On this ticket, we improve the message further.
-
-- First, these recommended packages don't actually help to avoid building any of the listed SPKGs. 
-
-  So it could be argued that `./configure` should recommend them separately. This is similar to #29372 "At the end of `configure`, show installation hints for disabled optional packages separately".
+- #29372 changed the message, showing installation hints for non-enabled optional packages separately.

-- Second, this message is printed regardless of whether `ffmpeg` et al. are installed because build/pkgs/_recommended/spkg-configure.m4 does not check anything.
+The message is printed regardless of whether `ffmpeg` et al. are installed because `build/pkgs/_recommended/spkg-configure.m4` does not check anything.

-  To this end, tests need to be added for the presence of the various recommended packages in the system. Either in `build/pkgs/_recommended/spkg-configure.m4` or in individual script packages similar to `build/pkgs/pandoc`.
+To this end, tests need to be added for the presence of the various recommended packages in the system. 
+
+**We do this by splitting out individual script packages, similar to the existing `build/pkgs/pandoc` and `build/pkgs/graphviz`.**

+

[seblabbe]

Description changed:

--- 
+++ 
@@ -19,7 +19,7 @@
 - #30910 replaced `texlive-generic-extra` by a package that exists in modern Debian distributions.
 - #31042 addresses `pandoc`.
 - #29372 changed the message, showing installation hints for non-enabled optional packages separately.
-
+- #32956 create dummy script packages for ffmpeg and imagemagick 

 The message is printed regardless of whether `ffmpeg` et al. are installed because `build/pkgs/_recommended/spkg-configure.m4` does not check anything.