Meta-ticket: Create upstream repositories, pip-installable packages for database packages

tobiasdiez commented 4 years ago

The Sage distribution contains a number of "database packages", many of which do not seem to have a "real upstream" (or installation procedures other than our scripts in build/pkgs).

We transform them into pip-installable packages and publish them to PyPI so that they can be installed using standard Python tools (and become Python dependencies of sagelib) and then discovered by libraries using standard Python facilities.

Role model: https://pypi.org/project/database-knotinfo/

For those that do not have a real upstream, we create separate git repositories in github.com/sagemath/ to serve as new upstream.

List of packages/tickets:

Architecture independent:

[ ] database_stein_watkins (#33881)
[ ] combinatorial_designs (single text file MOLS_table.txt), 8 kB, https://repology.org/project/sagemath-combinatorial-designs/versions
[ ] graphs (graphs.db, brouwer_srg_database.json, smallgraphs.txt, isgci_sage.xml), 336 kB; the SPKG.rst explains: The code used to parse the data from Andries E. Brouwer's website is available at https://github.com/nathanncohen/strongly_regular_graphs_database
[ ] polytopes_db (various files), 41 kB
[ ] polytopes_db_4d (various files), 878 MB

Prepared files:

[x] conway_polynomials (#32747, #36765)
[ ] elliptic_curves

New data sources:

[x] #30352 Interface to the KnotInfo and LinkInfo databases
[ ] database_graver_ppi - https://www.math.ucdavis.edu/~mkoeppe/art/ppi/index.html
[ ] https://mathdb.mathhub.info/
[ ] https://swmath.org/

Related (other stuff written by the Sage distribution into $SAGE_SHARE):

30306 - jupyter notebook related things
20080 - documentation
30885 - sage.features

Techniques:

importlib.resources - would help making a distribution zip_safe (#31306)
could consider advertise install location using a pkgconfig file: #30787 comment:24
Reduce local duplication of large database package when venvs are in use: https://pypi.org/project/pydupes/ (similar to rdfind)
Avoid duplicate installation by using PEP 660 editable wheels. Use a build backend that supports it: flit, pdm, hatchling, poetry, not setuptools. https://discuss.python.org/t/pep-660-and-setuptools/14855
However there are implementation restrictions in the interaction of editable wheels and implicit namespace packages

CC: @mkoeppe @jhpalmieri @kiwifb @isuruf @antonio-rojas @dimpase @slel @tobihan @jamesjer @soehms @videlec @williamstein @roed314

Component: build

Keywords: sd111

Issue created by migration from https://trac.sagemath.org/ticket/30914

mkoeppe commented 4 years ago

comment:1

Which files, specifically?

mkoeppe commented 4 years ago

comment:2

If you are talking about database packages such as build/pkgs/conway_polynomials, in particular those that do not seem to have a "real upstream" (or installation procedures other than our scripts in build/pkgs), then yes, I agree that a possible direction would be to create pip-installable packages for them and to publish them to PyPI. They should be maintained outside of the source tree, each in a separate git repository.

mkoeppe commented 4 years ago

comment:3

If this is what you had in mind, let's transform this ticket into a meta-ticket and get the downstream packagers of Sage on board -- they may have insights on the upstream status of some of these packages, which we may have forgotten about.

kiwifb commented 4 years ago

comment:4

Yes, a confirmation of which bits would be nice. If @mkoeppe is right you are talking about

combinatorial_designs
conway_polynomials
elliptic_curves
graphs
polytopes_db
plus some other optional packages They are indeed static but come in two flavors.

Architecture independent:

combinatorial_designs
graphs
polytopes_db which are really file copying.

Prepared files

conway_polynomials
elliptic_curves where the files are processed before install with some python code that may be depending on python or some other software.

conway_polynomials uses pickle.dump which could pose problems across python versions?

elliptic_curves is put into a SQL database with sqlite, this should be a stable format. Don't know if that's portable between, say, x86_64 and ppc64le?

mkoeppe commented 4 years ago

comment:5

OK, I've taken the liberty to rewrite the ticket description

mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -1,11 +1,11 @@
-Problem: While working on #30371, I noticed that most files in `local/share` are more or less static, but are generated during `make`.
-I was wondering how to streamline this process a bit in order to make modularization easier, and to make it possible to install sagelib in different venv and without a previous run of `bootstrap`/`make`.
+The Sage distribution contains a number of "database packages", many of which do not seem to have a "real upstream" (or installation procedures other than our scripts in build/pkgs).

-Options that I see:
-1. Checkin the current version of these files (at least the almost static databases etc) in say `src/share`. Then they only need to be generated anew for update due to changes upstream. So one can remove the corresponding sage packages from the `make` pipeline.
+We transform them into pip-installable packages and publish them to PyPI so that they can be installed using standard Python tools (and become Python dependencies of sagelib).

-2. Publish these dependencies to pypi, and install them to the current venv using requirements.txt and/or pipfile.
+For those that do not have a real upstream, we create separate git repositories in github.com/sagemath/ to serve as new upstream.

-3. Generate these files during a call of `src/setup.py`, and include them in this way in the current venv.
+List of packages/tickets:

-Any other ideas/preferences about how to handle these files?
+TBD
+
+

mkoeppe commented 4 years ago

comment:6

Replying to @kiwifb:

Prepared files

conway_polynomials

elliptic_curves where the files are processed before install with some python code that may be depending on python or some other software.

conway_polynomials uses pickle.dump which could pose problems across python versions?

elliptic_curves is put into a SQL database with sqlite, this should be a stable format. Don't know if that's portable between, say, x86_64 and ppc64le?

For these ones, we should make the actual source distribution architecture-independent. The architecture-dependent products could be built during setup.py - and published as wheels.

mkoeppe commented 4 years ago

comment:7

For example database_stein_watkins_mini's SPKG.rst lists http://modular.math.washington.edu/papers/stein-watkins/ as upstream or source, which is defunct

mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -6,6 +6,15 @@

 List of packages/tickets:

-TBD
+Architecture independent:
+
+- combinatorial_designs
+- graphs
+- polytopes_db
+
+Prepared files:
+
+- conway_polynomials
+- elliptic_curves

mkoeppe commented 4 years ago

comment:8

One thing that may need discussion is that for huge databases, one may want to avoid installing multiple copies

mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -18,3 +18,8 @@
 - elliptic_curves

+
+Related (other stuff written by the Sage distribution into $SAGE_SHARE): 
+- #30306 - jupyter notebook related things
+- #20080 - documentation
+

slel commented 4 years ago

comment:10

Replying to @mkoeppe:

For example database_stein_watkins_mini's SPKG.rst lists http://modular.math.washington.edu/papers/stein-watkins/ as upstream or source, which is defunct

It's at https://wstein.org/papers/stein-watkins/

tobiasdiez commented 4 years ago

comment:11

Replying to @kiwifb:

Yes, a confirmation of which bits would be nice. If @mkoeppe is right you are talking about ...

Yes, that's exactly what I meant! Thanks everybody for their input, that's already way better than what I've imagined originally.