search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.43k stars 479 forks source link

tox.ini, build/bin/write-dockerfile.sh: Add gentoo linux, add more gentoo packages #29105

Closed mkoeppe closed 4 years ago

mkoeppe commented 4 years ago

As a followup to #29053 and #29273, we

To test: tox -e docker-gentoo-standard

This is a part of:

Resources:

Depends on #29273

CC: @orlitzky @kiwifb @dimpase @sheerluck

Component: porting

Author: Andrey Belgorodski

Branch: 9959ee9

Reviewer: Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/29105

mkoeppe commented 4 years ago
comment:1

Creating a dockerfile for setting up sage prerequisites on a Gentoo "prefix" was previously discussed in #28905.

An attempt (incomplete) is at https://github.com/sagemath/sage-prod/files/10659346/Dockerfile-sage_gentoo.gz and would need help from Gentoo experts.

mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -1,6 +1,6 @@
 As a followup to #29053, in analogy to the work done for debian/fedora/arch/conda, we
 - add files `build/pkgs/SPKG/distros/gentoo.txt` to keep track of equivalent distribution packages
-- extend `build/bin/write-dockerfile` to generate a Dockerfile for a gentoo "prefix" build.
+- extend `build/bin/write-dockerfile` to generate a Dockerfile for a gentoo "prefix" build - for rigorous multi-platform testing.

 This is a part of:
 - #29060: Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages
kiwifb commented 4 years ago
comment:2

compiler can't create executables in a configure phase of gcc typically means the compiler can't find libgmp and al. Do you have a log for me to review? Prefix on ubuntu is definitely supported. The only thing I can think of is that the bootstrap usually object being run as root.

mkoeppe commented 4 years ago
comment:3

Thanks, yes, turns out this was not specific to the gentoo prefix installer. I had to add some more packages. Let me see how far I get with this.

mkoeppe commented 4 years ago

Attachment: gentoo-log-2.txt

mkoeppe commented 4 years ago
comment:4

Attachment: Dockerfile-sage_gentoo.gz

OK... this is where I got now. The ./bootstrap-prefix.sh script was claiming to bootstrap some packages - such as xz and patch - and succeeded with that; but then it was missing "xz" to unpack the Python sources. I think something is wrong there, but I added xz&patch in ubuntu. This is attached Dockerfile-sage_gentoo and log from 'docker build' with that.

As you can see in the log, now it's complaining about a module missing in the Python that it just built:

  File "/gentoo/var/tmp/python-3.6.10/Python-3.6.10/Lib/xml/parsers/expat.py", line 4, in <module>
    from pyexpat import *
ModuleNotFoundError: No module named 'pyexpat'
make: *** [install] Error 1
Makefile:1102: recipe for target 'install' failed
??? Python failed to install *sigh* continuing anyway
* Python bootstrapped
http://distfiles.gentoo.org/snapshots

and then dies with:

* Compiling prefix-portage
./configure --host=x86_64-pc-linux-gnu --prefix=/gentoo/tmp/usr --mandir=/gentoo/tmp/usr/share/man --infodir=/gentoo/tmp/usr/share/info --datadir=/gentoo/tmp/usr/share --sysconfdir=/gentoo/tmp/etc --localstatedir=/gentoo/tmp/var/lib --build=x86_64-pc-linux-gnu --with-offset-prefix=/gentoo/tmp --with-portage-user=root --with-portage-group=root --with-extra-path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... x86_64-pc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
....
checking whether /usr/bin/id is good enough... yes
checking for python... no
configure: error: no python found in your path
kiwifb commented 4 years ago
comment:5

xz is rearing up its head again. Yes it should be an early one. Because many more sources are in .xz I am not shocked that it should be a pre-requisite along with patch. The pyexpat stuff is weird. I'll do some experimenting later. I haven't really touched prefix since ppc64 was retired (because I didn't want or could maintain it long term).

mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -1,6 +1,18 @@
 As a followup to #29053, in analogy to the work done for debian/fedora/arch/conda, we
 - add files `build/pkgs/SPKG/distros/gentoo.txt` to keep track of equivalent distribution packages
-- extend `build/bin/write-dockerfile` to generate a Dockerfile for a gentoo "prefix" build - for rigorous multi-platform testing.
+- extend `build/bin/write-dockerfile` to 
+   - generate a Dockerfile using https://hub.docker.com/r/gentoo/stage3-amd64/tags
+   - generate a Dockerfile for a gentoo "prefix" build on top of some Linux distribution
+  for rigorous multi-platform testing.

 This is a part of:
 - #29060: Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages
+
+---
+
+Resources:
+- https://forums.gentoo.org/viewtopic-t-969278-start-0.html
+- cschwan/sage-on-gentoo: (Unofficial) Gentoo Overlay for Sage- and Sage-related ebuilds: https://github.com/cschwan/sage-on-gentoo
+- Project:Overlays/Old User Guide - Gentoo Wiki: https://wiki.gentoo.org/wiki/Project:Overlays/Old_User_Guide
+- Layman - Gentoo Wiki: https://wiki.gentoo.org/wiki/Layman
+
mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -1,6 +1,7 @@
-As a followup to #29053, in analogy to the work done for debian/fedora/arch/conda, we
-- add files `build/pkgs/SPKG/distros/gentoo.txt` to keep track of equivalent distribution packages
-- extend `build/bin/write-dockerfile` to 
+As a followup to #29053, in analogy to the work done for debian/fedora/arch/conda, we are adding files `build/pkgs/SPKG/distros/gentoo.txt` to keep track of equivalent distribution packages in #29273.
+
+In this ticket, 
+- we extend `build/bin/write-dockerfile` to 
    - generate a Dockerfile using https://hub.docker.com/r/gentoo/stage3-amd64/tags
    - generate a Dockerfile for a gentoo "prefix" build on top of some Linux distribution
   for rigorous multi-platform testing.
mkoeppe commented 4 years ago

Dependencies: #29273

mkoeppe commented 4 years ago
comment:9

This tickets needs help from someone who actually uses Gentoo. I found the instructions for installing stuff inadequate.

dimpase commented 4 years ago
comment:10

I use Gentoo, but neither prefix, not docker.

kiwifb commented 4 years ago
comment:11

I need to make time for this. I cannot give you a time frame.

mkoeppe commented 4 years ago
comment:12

Replying to @dimpase:

I use Gentoo, but [...] not docker.

After installing a docker client, it only takes 

docker run -it gentoo/stage3-amd64

This gives you root prompt. I would need instructions on what to do next to install relevant gentoo packages here.

kiwifb commented 4 years ago
comment:13

If I understand well you have a minimal gentoo image running now. What do you want to install? Do you have a list somewhere?

mkoeppe commented 4 years ago
comment:14

Replying to @kiwifb:

If I understand well you have a minimal gentoo image running now. What do you want to install? Do you have a list somewhere?

The equivalent of the lists in build/pkgs/debian-bootstrap.txt and build/pkgs/debian.txt (minimal requirements for bootstrapping and building) -- which would be added in #29273 as well since gentoo packages are being added there.

dimpase commented 4 years ago
comment:15

Replying to @mkoeppe:

Replying to @dimpase:

I use Gentoo, but [...] not docker.

After installing a docker client, it only takes 

docker run -it gentoo/stage3-amd64

This gives you root prompt. I would need instructions on what to do next to install relevant gentoo packages here.

I can try either this or Gentoo in lxc container, if such a thing exists somewhere.

mkoeppe commented 4 years ago
comment:16

Docker please because that's what my tox.ini infrastructure uses. If we want to do the same with lxc instead of docker, why not, but perhaps one step at a time?

mkoeppe commented 4 years ago
comment:17

I have created ticket #29283 for the goal of lxc containers for testing, in case someone wants to work on that.

mkoeppe commented 4 years ago
comment:21

Starting points for people who want to help with this:

sheerluck commented 4 years ago
comment:22

When I run tox -e docker-gentoo I get half-baked sage/.tox/docker-gentoo/Dockerfile, I need a way to shove the hand-crafted Dockerfile instead (for debugging purposes). What is the best way to do it?

mkoeppe commented 4 years ago
comment:23

Does your handcrafted Dockerfile work with docker build already?

mkoeppe commented 4 years ago
comment:24

You could insert a line before # From https://hub.docker.com/r/multiarch/ubuntu-core/ that says

    docker-gentoo:  bash -c 'whatever command to create a handcrafted {envdir}/Dockerfile'
sheerluck commented 4 years ago
comment:25

So right after

docker:        bash -c 'build/bin/write-dockerfile.sh  ... > {envdir}/Dockerfile'

Thank you!

sheerluck commented 4 years ago

Attachment: docker-gentoo-installed.txt

docker-gentoo-installed.txt

sheerluck commented 4 years ago
comment:26

The work is still in progress. I managed to install 555 packages.

mkoeppe commented 4 years ago
comment:27

Nice

sheerluck commented 4 years ago
comment:28
real    14m14.888s
user    98m59.353s
sys     3m4.286s
SAGE_CHECK=warn, so scanning the log files. This may take a few seconds.
Sage build/upgrade complete!
make[1]: Leaving directory '/sage'
Removing intermediate container e0e5bbfb22ea
 ---> 14316b24894b
[Warning] One or more build-args [TARGETS_OPTIONAL] were not consumed
Successfully built 14316b24894b
Successfully tagged sage-docker-gentoo-standard-with-targets:9.2.beta1-dirty
_________________________ summary __________________________
  docker-gentoo-standard: commands succeeded
  congratulations :)

1) run docker build -t fbissey/sage-on-gentoo-stage4:20.6 -t fbissey/sage-on-gentoo-stage4:latest -f Dockerfile_gentoo-20.06-gcc9-py37 .

2) patch tox.ini and write-dockerfile.sh

sheerluck commented 4 years ago

Attachment: tox.ini.patch.gz

sheerluck commented 4 years ago

Attachment: write-dockerfile.sh.patch.gz

mkoeppe commented 4 years ago
comment:29

Attachment: Dockerfile_gentoo-20.06-gcc9-py37.gz

https://hub.docker.com/r/fbissey/sage-on-gentoo-stage/tags ... this does not seem to exist yet?

sheerluck commented 4 years ago
comment:30

Replying to @mkoeppe:

https://hub.docker.com/r/fbissey/sage-on-gentoo-stage/tags ... this does not seem to exist yet?

It does not exist. Someone have to decide where to store an image. I have no idea, my docker-fu is weak.

mkoeppe commented 4 years ago
comment:31

Why not create an account on docker hub? Next step: https://docs.docker.com/docker-hub/builds/

sheerluck commented 4 years ago
comment:32

I am working on Dockerfile_gentoo-20.06-gcc9-py37​ to make image size much much smaller :(

sheerluck commented 4 years ago
comment:33

docker pull sheerluck/sage-on-gentoo-stage4:latest

sheerluck commented 4 years ago
comment:34
$ docker run -it sheerluck/sage-on-gentoo-stage4 /bin/bash
cf6f39e01c4a / # ipython --no-banner

In [1]: from os import walk
   ...: pth, ext = "/var/db/pkg", ".ebuild"
   ...: lex, res = len(ext), []
   ...: for _, _, u in walk(pth):
   ...:     res += [x[:-lex] for x in u if x.endswith(ext)]
   ...: for  pkg  in sorted(res): print(pkg)
Alnuth-3.1.1
App-pwhich-1.150.0
Authen-SASL-2.160.0-r1
...
zn_poly-0.9.2
zope-interface-4.7.1
zstd-1.4.4-r4
zziplib-0.13.69-r1
mkoeppe commented 4 years ago
comment:35

Awesome!

mkoeppe commented 4 years ago
comment:36

Could you put the changes to the sage scripts on a git branch please?

sheerluck commented 4 years ago
comment:37
$ git push trac HEAD:u/gh-sheerluck/gentoo
...
To trac.sagemath.org:sage.git
 * [new branch]            HEAD -> u/gh-sheerluck/gentoo
mkoeppe commented 4 years ago

Branch: u/gh-sheerluck/gentoo

mkoeppe commented 4 years ago

New commits:

25d0579write-dockerfile.sh is extended to support gentoo
mkoeppe commented 4 years ago

Commit: 25d0579

mkoeppe commented 4 years ago
comment:39
sed: ./build/pkgs/gentoo.txt: No such file or directory
sed: ./build/pkgs/gentoo-bootstrap.txt: No such file or directory
mkoeppe commented 4 years ago
comment:40

Also, at the end of configure, I see:

configure: notice: the following SPKGs did not find equivalent system packages: cbc eclib gp2c isl lcalc libsemigroups pandoc pari pari_elldata pari_galdata pari_galpol pari_nftables pari_seadata pari_seadata_small perl_cpan_polymake_prereq perl_term_readline_gnu r yasm
checking for the package system in use... gentoo
configure: hint: installing the following system packages is recommended and may avoid building some of the above SPKGs from source:
configure: # install the following packages: sci-mathematics/eclib[flint] sci-mathematics/gp2c dev-libs/isl app-text/pandoc sci-mathematics/pari sci-mathematics/pari-data XML-Writer XML-LibXML XML-LibXSLT File-Slurp dev-perl/Term-ReadLine-Gnu JSON SVG dev-perl/MongoDB dev-perl/Term-ReadLine-Gnu dev-lang/R

Could you update build/bin/sage-print-system-package-command so that an actual installation command is displayed?

sheerluck commented 4 years ago
comment:41

I added gentoo.txt and gentoo-bootstrap.txt and updated sage-print-system-package-command and I am going to refresh sheerluck/sage-on-gentoo-stage4:latest with additional packages (except pandoc) and after testing I will push in u/gh-sheerluck/gentoo branch

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 25d0579 to c55cfa9

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

c55cfa9sage-print-system-package-command updated
sheerluck commented 4 years ago
comment:43

Now configure says "configure: $ sudo emerge sci-mathematics/eclib[flint] sci-mathematics/gp2c app-text/pandoc sci-mathematics/pari sci-mathematics/pari-data XML-Writer XML-LibXML XML-LibXSLT File-Slurp dev-perl/Term-ReadLine-Gnu JSON SVG dev-perl/MongoDB dev-lang/R" although everything is installed in sheerluck/sage-on-gentoo-stage4:latest (except pandoc)

mkoeppe commented 4 years ago
comment:44

For sci-mathematics/eclib[flint], please check whether this syntax does whatever was intended there. See previous discussion in #29273.

config.log should give information on why a particular package was rejected by configure.

sheerluck commented 4 years ago
comment:45

Reading config.log is fascinating.

For example, "cbc >= 2.9.4" might refer to sci-libs/coinor-cbc-2.10.5 (https://github.com/coin-or/Cbc)

WIP