search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.4k stars 474 forks source link

upgrade lrcalc to 2.1 #31355

Closed edd8e884-f507-429a-b577-5d554626c0fe closed 2 years ago

edd8e884-f507-429a-b577-5d554626c0fe commented 3 years ago

lrcalc 2.1 is out https://bitbucket.org/asbuch/lrcalc/src/master/ChangeLog

Instead of rewriting Sage's Python bindings to support the new API, we replace them with a wrapper around the new upstream provided bindings

Upstream:

lrcalclib-2.1.tar.gz (lrcalc): https://sites.math.rutgers.edu/~asbuch/lrcalc/lrcalc-2.1.tar.gz lrcalc-2.1.tar.gz (lrcalc_python): https://pypi.io/packages/source/l/lrcalc/lrcalc-2.1.tar.gz

CC: @antonio-rojas @anneschilling @fchapoton @tscrim @thierry-FreeBSD @orlitzky @mkoeppe @kiwifb @asbuch @slel

Component: packages: optional

Keywords: upgrade, lrcalc

Author: Antonio Rojas, Matthias Koeppe

Branch/Commit: de4fe09

Reviewer: Matthias Koeppe, Travis Scrimshaw

Issue created by migration from https://trac.sagemath.org/ticket/31355

edd8e884-f507-429a-b577-5d554626c0fe commented 3 years ago

Description changed:

--- 
+++ 
@@ -1,2 +1,2 @@
-lrslib 2.0 is out https://bitbucket.org/asbuch/lrcalc/src/master/ChangeLog
+lrcalc 2.0 is out https://bitbucket.org/asbuch/lrcalc/src/master/ChangeLog
edd8e884-f507-429a-b577-5d554626c0fe commented 3 years ago

Branch: u/tmonteil/upgrade_lrcalc_to_2_0

edd8e884-f507-429a-b577-5d554626c0fe commented 3 years ago

Commit: 850f9bc

edd8e884-f507-429a-b577-5d554626c0fe commented 3 years ago
comment:4

I get the following error when doctesting:

ImportError: /opt/sagemath/sage/local/lib/python3.7/site-packages/sage/libs/lrcalc/lrcalc.cpython-37m-x86_64-linux-gnu.so: undefined symbol: st_new

New commits:

850f9bc#31355 : upgrade lrcalc to 2.0
antonio-rojas commented 3 years ago
comment:5

This has major API changes. Large parts of lrcalc.pyx needs to be rewritten because the used functions have been removed or modified.

edd8e884-f507-429a-b577-5d554626c0fe commented 3 years ago
comment:6

I see, any help welcome !

orlitzky commented 3 years ago
comment:8

The existing spkg-configure.m4 needs to be updated to detect v2.0 (once it actually works), too.

nthiery commented 3 years ago
comment:9

Some information from Anders:

I recently put out a new version of my lrcalc program, much of it is rewritten, and some of the most important functions are close to twice as fast as before.  Some special cases have also been optimized to run a lot faster.

I have included some (pure) Python3 bindings in the repository, so I think this update will be easy.  But I don't understand Sage well enough to do it myself.

For reference, some pointers:

asbuch commented 3 years ago
comment:10

The Python bindings are only in the repository and a bit raw, so probably only useful as examples. But I expect they are easier to start from than the .h files.

antonio-rojas commented 3 years ago

Changed branch from u/tmonteil/upgrade_lrcalc_to_2_0 to u/arojas/upgrade_lrcalc_to_2_0

antonio-rojas commented 3 years ago
comment:12

Replying to @asbuch:

The Python bindings are only in the repository and a bit raw, so probably only useful as examples. But I expect they are easier to start from than the .h files.

Actually, they are almost equivalent to the current bindings in Sage. The attached branch replaces lrcalc.pyx with a thin wrapper over lrcalc's bindings.

It mostly just translates the output from lrcalc to sage notation. The only function of lrcalc.pyx that isn't straghtforward to port is quantum multiplication: I found no way to recover the q-grading from lrcalc's output, so this is reimplemented in the patch.

The branch needs an update of lrcalc to include python bindings.

New commits:

6596bbdReplace lrcalc.pyx with a wrapper over lrcalc's own Python bindings
antonio-rojas commented 3 years ago

Changed commit from 850f9bc to 6596bbd

antonio-rojas commented 3 years ago
comment:13

Replying to @asbuch:

The Python bindings are only in the repository and a bit raw, so probably only useful as examples. But I expect they are easier to start from than the .h files.

Since you are here and I can not find a bug tracker for lrcalc: the setup.py is missing metadata, so the egg for lrcalc gets installed as UNKNOWN-0.0.0.0

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

0a9490aMerge branch 'develop' of git://git.sagemath.org/sage into t/31355/upgrade_lrcalc_to_2_0
26f7f26Fix oversight in q_degrees formula
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 6596bbd to 26f7f26

asbuch commented 3 years ago
comment:15

If my bindings are used, then that will make it easier to keep things compatible if I make more changes to the C interface. I should be able to keep the Python interface stable, at least up to reordering of output.

Quantum function: I will add a function mult_quantum_pairs that returns a dict from ((partition),d) to integers, you can switch to that if you want.

setup.py: Does the following include the tags you are looking for? Suggestions welcome!

setup(name='lrcalc',
    version='2.0.0',
    description='Littlewood-Richardson Calculator',
    long_description='Littlewood-Richardson Calculator',
    url='https://math.rutgers.edu/~asbuch/lrcalc',
    author='Anders Skovsted Buch',
    license='GPL3',
    ext_modules = cythonize([
        Extension("lrcalc", ["lrcalc.pyx"],
                  libraries=["lrcalc"])
    ])
)
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 26f7f26 to 627192a

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

627192aUse Sage integers in results
antonio-rojas commented 3 years ago
comment:17

Replying to @asbuch:

setup.py: Does the following include the tags you are looking for? Suggestions welcome!

setup(name='lrcalc',
    version='2.0.0',
    description='Littlewood-Richardson Calculator',
    long_description='Littlewood-Richardson Calculator',
    url='https://math.rutgers.edu/~asbuch/lrcalc',
    author='Anders Skovsted Buch',
    license='GPL3',
    ext_modules = cythonize([
        Extension("lrcalc", ["lrcalc.pyx"],
                  libraries=["lrcalc"])
    ])
)

That works, thanks. Also, although maybe it is a little bit too late for that, it would be good to bump the soversion of the C shared library since the API changed.

asbuch commented 3 years ago
comment:18

Replying to @antonio-rojas:

Replying to @asbuch:

setup.py: Does the following include the tags you are looking for? Suggestions welcome!

setup(name='lrcalc',
    version='2.0.0',
    description='Littlewood-Richardson Calculator',
    long_description='Littlewood-Richardson Calculator',
    url='https://math.rutgers.edu/~asbuch/lrcalc',
    author='Anders Skovsted Buch',
    license='GPL3',
    ext_modules = cythonize([
        Extension("lrcalc", ["lrcalc.pyx"],
                  libraries=["lrcalc"])
    ])
)

That works, thanks. Also, although maybe it is a little bit too late for that, it would be good to bump the soversion of the C shared library since the API changed.

I changed the version to "2.0.0" and thought that was in line with this page. Please let me know if I am doing something wrong.

antonio-rojas commented 3 years ago
comment:19

Replying to @asbuch:

I changed the version to "2.0.0" and thought that was in line with this page. Please let me know if I am doing something wrong.

You need to change the -version-info in https://bitbucket.org/asbuch/lrcalc/src/master/src/Makefile.am#lines-23

asbuch commented 3 years ago
comment:20

Replying to @antonio-rojas:

Replying to @asbuch:

I changed the version to "2.0.0" and thought that was in line with this page. Please let me know if I am doing something wrong.

You need to change the -version-info in https://bitbucket.org/asbuch/lrcalc/src/master/src/Makefile.am#lines-23

I see, thanks! According to conventions, is that line in Makefile.am independent from this line in configure.am:

AC_INIT([lrcalc],[2.0.0],[asbuch at math rutgers edu])
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

52c120fUse new mult_quantum degrees option in lrcalc 2.1 for quantum multiplication
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 627192a to 52c120f

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

e13f155Upgrade lrcalc to 2.1 and install python bindings
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 52c120f to e13f155

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from e13f155 to b2d2844

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

b2d2844Account for the possibility of different degree terms indexed by the same partition
antonio-rojas commented 3 years ago
comment:24

spkg-configure needs to be updated to check whether the python bindings are installed, and build sage's spkg otherwise.

antonio-rojas commented 3 years ago

Changed author from Thierry Monteil to Thierry Monteil, Antonio Rojas

asbuch commented 3 years ago
comment:27

Sorry for the moving target. However, 2.1 should bring only more speed and no new problems.

The mult_quantum() function now takes an optional boolean arguments "degrees" that causes it to replace each key part with (part,d).

mkoeppe commented 3 years ago
comment:28

Replying to @antonio-rojas:

spkg-configure needs to be updated to check whether the python bindings are installed, and build sage's spkg otherwise.

We cannot use Python packages from the system, see #29023.

An SPKG providing both a library and Python bindings to it should be split into 2 SPKGs.

mkoeppe commented 3 years ago
comment:29

Anders, the Python package should declare its build dependencies (in particular Cython) by putting a pyproject.toml file next to the setup.py; see https://snarky.ca/what-the-heck-is-pyproject-toml/

antonio-rojas commented 3 years ago
comment:30

Replying to @mkoeppe:

Replying to @antonio-rojas:

spkg-configure needs to be updated to check whether the python bindings are installed, and build sage's spkg otherwise.

We cannot use Python packages from the system, see #29023.

An SPKG providing both a library and Python bindings to it should be split into 2 SPKGs.

Then maybe sage shouldn't support using system lrcalc 2.x for now, the bindings and the C library should obviously be kept at the same version.

In any case, the sagelib side should be ready now, please someone else take care of the build side.

mkoeppe commented 3 years ago
comment:31

Whether we support system lrcalc or not, non-Python and Python installations need to be split into two separate SPKGs (they can share the same tarball).

asbuch commented 3 years ago
comment:32

Is this the pyproject.toml that you need? If so, then I can add that.

[build-system]
requires = ["setuptools", "cython"]
build-backend = "setuptools.build_meta"

Regarding splitting into SPKGs, I am not familiar with the details of Python's package system. Maybe one day I will read up on this and make it possible to say pip install lrcalc, but this is not likely to happen soon. One way to proceed for now is to copy my files liblrcalc.pxd and lrcalc.pyx to the Sage repository next to Nicolas' files. Changes to liblrcalc's interface are not likely to happen soon, but even if that happened, Sage could be brought up-to-date with another copy command. Not perfect, but maybe still a small improvement to the way lrcalc has been integrated in the past?

mkoeppe commented 3 years ago
comment:33

Replying to @asbuch:

Is this the pyproject.toml that you need? If so, then I can add that.

[build-system]
requires = ["setuptools", "cython"]
build-backend = "setuptools.build_meta"

Yes, that's good; best to add "wheel" in requires too.

Regarding splitting into SPKGs, I am not familiar with the details of Python's package system.

This is actually Sage-specific, not related to Python packaging, so not a task for you as the upstream author.

"Best practices" on your side would be to create a source distribution for the Python library, using setup.py sdist and to upload it to PyPI. On the Sage packaging side, we can then use that as our upstream for a lrcalc_python SPKG.

mkoeppe commented 3 years ago
comment:34

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from b2d2844 to 1153526

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

4acca82Merge branch 'develop' of git://git.sagemath.org/sage into t/31355/upgrade_lrcalc_to_2_0
464d5ceMove lrcalc python bindings to a separate spkg
346b76bAdmit only system lrcalc v2
1153526Document _lrcalc_dict_to_sage
antonio-rojas commented 3 years ago
comment:36

Trying to move this forward

antonio-rojas commented 3 years ago

Description changed:

--- 
+++ 
@@ -1,2 +1,3 @@
-lrcalc 2.0 is out https://bitbucket.org/asbuch/lrcalc/src/master/ChangeLog
+lrcalc 2.1 is out https://bitbucket.org/asbuch/lrcalc/src/master/ChangeLog

+Instead of rewriting Sage's Python bindings to support the new API, we replace them with a wrapper around the new upstream provided bindings
mkoeppe commented 3 years ago
comment:38

-1 on using a Python package that is not on PyPI

mkoeppe commented 3 years ago
comment:39

Setting a new milestone for this ticket based on a cursory review.

dimpase commented 3 years ago
comment:40

Replying to @mkoeppe:

-1 on using a Python package that is not on PyPI

we have to move forward here, still. So we can either use the non-PyPI package, or facilitate a release on PyPI...

edd8e884-f507-429a-b577-5d554626c0fe commented 3 years ago
comment:41

Replying to @dimpase:

Replying to @mkoeppe:

-1 on using a Python package that is not on PyPI

we have to move forward here, still. So we can either use the non-PyPI package, or facilitate a release on PyPI...

Let me suggest to postpone helping upstream to use PyPI to another ticket.

mkoeppe commented 3 years ago
comment:42

It would block the Sage library (sagemath-standard) from being pushed to PyPI, so no, that's not a viable approach.

mkoeppe commented 3 years ago
comment:43

Replying to @dimpase:

facilitate a release on PyPI...

Facilitated now by providing this link: https://realpython.com/pypi-publish-python-package/#publishing-to-pypi

dimpase commented 3 years ago
comment:44

Someone would probably need to provide a PR for upstream, not just an RTFM link.

mkoeppe commented 3 years ago
comment:45

No PR is needed, the upstream maintainer really just needs to type python3 setup.py sdist && twine upload dist/*