search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.45k stars 482 forks source link

build/pkgs: Split out spkg liblzma from xz #30948

Closed mkoeppe closed 3 years ago

mkoeppe commented 3 years ago

(from #30559)

$ grep xz build/pkgs/*/dependencies
build/pkgs/cmake/dependencies:curl zlib bzip2 xz
build/pkgs/gcc/dependencies:| $(MP_LIBRARY) mpfr mpc zlib xz
build/pkgs/gdb/dependencies:mpfr zlib ncurses $(PYTHON) xz
build/pkgs/gfortran/dependencies:| $(MP_LIBRARY) mpfr mpc zlib xz
build/pkgs/p_group_cohomology/dependencies:$(PYTHON)... | $(PYTHON_TOOLCHAIN) matplotlib gap xz $(SAGERUNTIME) ipywidgets
build/pkgs/python3/dependencies:zlib readline sqlite libpng bzip2 xz libffi
build/pkgs/r/dependencies:$(BLAS) gfortran iconv readline bzip2 xz pcre curl | pkgconf

In this list, cmake, r, python3 all depend on liblzma. The others are merely build-time dependencies on the xz binary (for unpacking the tarball in sage_bootstrap.uncompress.tar_file - when sage-bootstrap-python is < 3.3 - see https://docs.python.org/3/library/tarfile.html) and have nothing to do with the shared library.

The new liblzma package is a script package without installation steps; it delegates the actual installation to the xz spkg. This follows the same approach as the libnauty/nauty packages.

Depends on #31409 Depends on #30350

CC: @dimpase @jhpalmieri

Component: packages: standard

Author: Matthias Koeppe

Branch/Commit: 1fb1fa9

Reviewer: John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/30948

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -1,3 +1,15 @@
 (from #30559)

+```
+$ grep xz build/pkgs/*/dependencies
+build/pkgs/cmake/dependencies:curl zlib bzip2 xz
+build/pkgs/gcc/dependencies:| $(MP_LIBRARY) mpfr mpc zlib xz
+build/pkgs/gdb/dependencies:mpfr zlib ncurses $(PYTHON) xz
+build/pkgs/gfortran/dependencies:| $(MP_LIBRARY) mpfr mpc zlib xz
+build/pkgs/p_group_cohomology/dependencies:$(PYTHON)... | $(PYTHON_TOOLCHAIN) matplotlib gap xz $(SAGERUNTIME) ipywidgets
+build/pkgs/python3/dependencies:zlib readline sqlite libpng bzip2 xz libffi
+build/pkgs/r/dependencies:$(BLAS) gfortran iconv readline bzip2 xz pcre curl | pkgconf
+```
+In this list, `cmake`, `r`, `python3` all depend on `liblzma`. 
+The others are merely build-time dependencies on the `xz` binary (for unpacking the tarball in `sage_bootstrap.uncompress.tar_file` - when `sage-bootstrap-python` is < 3.3 - see https://docs.python.org/3/library/tarfile.html) and have nothing to do with the shared library.
jhpalmieri commented 3 years ago
comment:2

Python should be able handle uncompressing .tar.xz files, so maybe we should modify the SageTarXZFile class in sage_bootstrap.uncompress.tar_file: only use it if Python can't already handle .xz files.

jhpalmieri commented 3 years ago
comment:3

We could test

hasattr(tarfile.TarFile, 'xzopen')

for example.

jhpalmieri commented 3 years ago
comment:4

I guess this shouldn't be necessary: sage_bootstrap.uncompress.action.open_archive should check to see if tarfile.is_tarfile(FILE) is True first, and it should be for xz files with a supported version of Python. So I still don't know what happened in Volker's case: using an old Python?

mkoeppe commented 3 years ago
comment:5

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

mkoeppe commented 3 years ago
comment:6

If r is coming from the system or --disable-r (#31409) is used, then we can drop the SAGE_SPKG_DEPCHECK for xz in python3/spkg-configure.m4 because then there is no package depending on liblzma that will run in the same process as python3. (cmake depends on liblzma but it does not run in the same process.)

mkoeppe commented 3 years ago

Dependencies: #31409

mkoeppe commented 3 years ago

Branch: u/mkoeppe/build_pkgs__split_out_spkg_liblzma_from_xz

mkoeppe commented 3 years ago

Author: Matthias Koeppe

mkoeppe commented 3 years ago

Commit: 546df52

mkoeppe commented 3 years ago

New commits:

c504bb9build/pkgs: Split out spkg liblzma from xz
f83aac5build/pkgs/{cmake,r}/dependencies: Depend on liblzma, not xz
9e1e643build/pkgs/python3/dependencies: Add liblzma
fe36a64build/pkgs/r/spkg-configure.m4: Change DEPCHECK from xz to liblzma
212b03ebuild/pkgs/python3/spkg-configure.m4: Change DEPCHECK from xz to liblzma
546df52build/pkgs/xz/spkg-configure.m4: Set sage_require_xz=no if sage-bootstrap-python is new enough
mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -13,3 +13,5 @@
 In this list, `cmake`, `r`, `python3` all depend on `liblzma`. 
 The others are merely build-time dependencies on the `xz` binary (for unpacking the tarball in `sage_bootstrap.uncompress.tar_file` - when `sage-bootstrap-python` is < 3.3 - see https://docs.python.org/3/library/tarfile.html) and have nothing to do with the shared library.

+The new `liblzma` package is a script package without installation steps; it delegates the actual installation to the `xz` spkg. This follows the same approach as the `libnauty`/`nauty` packages.
+
mkoeppe commented 3 years ago
comment:10

The changes discussed in comment:2 to comment:4 still need to be implemented

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

2b4b142build/pkgs: Split out spkg liblzma from xz
5137ae9build/pkgs/{cmake,r}/dependencies: Depend on liblzma, not xz
462fd33build/pkgs/python3/dependencies: Add liblzma
ba6fdcdbuild/pkgs/r/spkg-configure.m4: Change DEPCHECK from xz to liblzma
bba3a93build/pkgs/python3/spkg-configure.m4: Change DEPCHECK from xz to liblzma
bf500bebuild/pkgs/xz/spkg-configure.m4: Set sage_require_xz=no if sage-bootstrap-python is new enough
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 546df52 to bf500be

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

bc01006build/pkgs/xz: Update to 5.2.5, add upstream_url
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from bf500be to bc01006

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

009256ebuild/pkgs/liblzma/dependencies: Update
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from bc01006 to 009256e

jhpalmieri commented 3 years ago
comment:15

Should there be a symlink build/pkgs/liblzma/package-version.txt -> ../xz/package-version.txt? Or is it okay to omit the version for lzma?

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 009256e to de28c72

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

de28c72build/pkgs/liblzma: Add symlink for package-version.txt
mkoeppe commented 3 years ago
comment:17

It's ok to omit it (it is using "none" in this case), but I agree that it's probably nicer with a version

jhpalmieri commented 3 years ago
comment:18

Thank you, looks good.

jhpalmieri commented 3 years ago

Reviewer: John Palmieri

mkoeppe commented 3 years ago
comment:19

Thanks!

vbraun commented 3 years ago
comment:20

Merge conflict

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

8ba1d79trac 30350: remove the ATLAS package
69db839trac 30350: remove "--with-blas=" option to ./configure
272424atrac 30350: remove remaining bits in build/pkgs/atlas/distros
1fb1fa9Merge #30350
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from de28c72 to 1fb1fa9

mkoeppe commented 3 years ago

Changed dependencies from #31409 to #31409, #30350

vbraun commented 3 years ago

Changed branch from u/mkoeppe/build_pkgs__split_out_spkg_liblzma_from_xz to 1fb1fa9