make sure all optional packages work in the git repository

[ohanar]

we support them, so we need to make sure they will continue to work

Script to systematically try out all optional spkgs: http://boxen.math.washington.edu/home/vbraun/Sage/Testing/test_optional_packages.sh

The optional databases will be dealt with here, other packages should make separate tickets. Use the following new "upstream" tarballs:

• http://boxen.math.washington.edu/home/vbraun/upstream/database_cremona_ellcurve-20121022.tar.bz2

Leave for later, don't work and no SPKG.txt:

• cunningham_tables
• database_jones_numfield
• database_kohel
• database_odlyzko_zeta
• database_sloane_oeis
• database_stein_watkins_mini.p0

Move to experimental (broken with or without git, probably for a long time):

• fricas (doesn't build with our ecl)
• fricas-aldor (dito)
• libcocoa (doesn't build with our gmp/mpir)

Delete for other reasons:

• jsmath_image_fonts (we switched to mathjax)

Depends on #15172 Depends on #15261 Depends on #15263 Depends on #15370 Depends on #15302 Depends on #15814

CC: @haraldschilly

Component: packages: optional

Author: Volker Braun

Branch/Commit: u/vbraun/optional_databases @ 5b72359

Reviewer: Volker Braun

Issue created by migration from https://trac.sagemath.org/ticket/14962

[ohanar]

Branch: u/ohanar/optional

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Recent commits:
[changeset:acbbe0d]	Merge branch 'build_system' into optional
[changeset:2535903]	[FIXUP] optional: add git specific files
[changeset:c463e93]	[CLEANUP] remove hgignores
[changeset:8bb933a]	Merge in optional packages
[changeset:4ea72f0]	Trac #14909: Add new dependencies of the HAP package

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Commit: acbbe0d

[ohanar]

Changed branch from u/ohanar/optional to u/ohanar/test

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from acbbe0d to cfd9dd1

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from cfd9dd1 to cf45f45

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from cf45f45 to 728e542

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 728e542 to cf45f45

[vbraun]

Changed commit from cf45f45 to none

[vbraun]

Dependencies: #15172

[vbraun]

comment:8

4ti2 is currently broken (not related to the git transition) but works with the update at #15172.

[vbraun]

Description changed:

--- 
+++ 
@@ -1 +1,3 @@
 we support them, so we need to make sure they will continue to work
+
+Script to systematically try out all optional spkgs: http://boxen.math.washington.edu/home/vbraun/Sage/Testing/test_optional_packages.sh

[vbraun]

Changed branch from u/ohanar/test to none

[vbraun]

Changed dependencies from #15172 to #15172, #15258

[vbraun]

comment:10

15258 fails, because it implements a dependency check in the spkg-install using newest-version. How should optional spkgs handle dependencies when they transition to the new layout, by adding to deps? Or put the dependency rule in a <packagename>/deps file that is included from deps? Or no dependency management?

[ohanar]

comment:11

For right now I would say no dependency management -- any other solution at this point in time would require messing with the build system, which I don't want to do at the moment (so that we can get through this git transition).

On top of our current faux package manager I would say the best solution would be to replace (at least most of) deps with <packagename>/deps, but I'm not particularly interested in continuing the development of the current build system.

[vbraun]

comment:12

The following spkgs failed to install:

brian
cunningham_tables
database_cremona_ellcurve
database_jones_numfield
database_kohel
database_odlyzko_zeta
database_sloane_oeis
database_stein_watkins_mini.p0
database_symbolic_data
fricas
fricasaldor
jsmath_image_fonts
libcocoa
libtheora
mpi4py
openssl
p_group_cohomology
PyQt_x11
TOPCOM
valgrind

Logfiles are here: http://boxen.math.washington.edu/home/vbraun/Sage/Testing/optional/

[vbraun]

Branch: u/vbraun/optional_databases

[vbraun]

Commit: 5b72359

[vbraun]

New commits:

[changeset:5b72359]

switched the large Cremona ellcurve database to the new git directory layout

[vbraun]

Author: Volker Braun

[vbraun]

comment:14

SAGE_DATA is now SAGE_LOCAL/share, for the record.

[vbraun]

Description changed:

--- 
+++ 
@@ -1,3 +1,6 @@
 we support them, so we need to make sure they will continue to work

 Script to systematically try out all optional spkgs: http://boxen.math.washington.edu/home/vbraun/Sage/Testing/test_optional_packages.sh
+
+The optional databases will be dealt with here, other packages should make separate tickets. Use the following new "upstream" tarballs:
+* http://boxen.math.washington.edu/home/vbraun/upstream/database_cremona_ellcurve-20121022.tar.bz2

[vbraun]

comment:16

Also, extra_docs barfs the extra docs into SAGE_ROOT.

[kcrisman]

comment:17

I am not online with the git layout yet (need to attend a tutorial, at the least, though Volker's video was a helpful start) so I can't help with this writing or reviewing yet BUT I have diagnosed the brian problem.

./spkg-install: line 10: cd: /mnt/storage2TB/vbraun-data/Sage/spkg/standard/: No such file or directory
./spkg-install: line 10: ./newest_version: No such file or directory
Failed to find numpy.  Please install the numpy spkg.

There are four tests of the following kind:

NUMPY=`cd $SAGE_ROOT/spkg/standard/; ./newest_version numpy`

So where would we check the "newest version" now? That's all one would need to do, or even check "presence" if the spkg placeholder txt files are no longer there. (That said, I'm not sure if those checks are even really necessary, which would also fix it, I'm sure. But if the fix is easy, then we should try that first.)

[vbraun]

comment:18

The brian spkg just needs to strip out all the dependency checking, then it should work. You can still make spkgs as in the old workflow. Git installation instructions are, by now, easy:

git clone git://github.com/sagemath/sage.git
cd sage
make

[kcrisman]

comment:19

The brian spkg just needs to strip out all the dependency checking, then it should work.

As I said:

That said, I'm not sure if those checks are even really necessary, which would also fix it, I'm sure.

I just don't remember why that was introduced. #9675 doesn't make it clear, so when I open this ticket, I'll cc: the original author. Otherwise I agree we should get rid of this. I'll update the patch situation as well and perhaps upgrade Brian if I find time for all this- should be able to in the next couple days.

You can still make spkgs as in the old workflow. Git installation instructions are, by now, easy:

Good to know! And that includes all the spkg stuff.

[kcrisman]

comment:20

I just don't remember why that was introduced. #9675 doesn't make it clear, so when I open this ticket, I'll cc: the original author. Otherwise I agree we should get rid of this. I'll update the patch situation as well and perhaps upgrade Brian if I find time for all this- should be able to in the next couple days.

This is now #15261.

[vbraun]

Changed dependencies from #15172, #15258 to #15172, #15261

[vbraun]

comment:22

• mpi4py actually works, just needs openmpi libraries/compiler wrapper installed

[vbraun]

Description changed:

--- 
+++ 
@@ -4,3 +4,20 @@

 The optional databases will be dealt with here, other packages should make separate tickets. Use the following new "upstream" tarballs:
 * http://boxen.math.washington.edu/home/vbraun/upstream/database_cremona_ellcurve-20121022.tar.bz2
+
+Move to experimental (broken with or without git, probably for a long time):
+* fricas (doesn't build with our ecl)
+* fricas-aldor (dito)
+* libcocoa (doesn't build with our gmp/mpir)
+
+Delete because no copyright information (ask again on sage-devel first):
+* cunningham_tables
+* database_jones_numfield
+* database_kohel
+* database_odlyzko_zeta
+* database_sloane_oeis
+* database_stein_watkins_mini.p0
+
+Delete for other reasons:
+* jsmath_image_fonts (we switched to mathjax)
+

[vbraun]

Changed dependencies from #15172, #15261 to #15172, #15261, #15263, #15302

[ohanar]

Changed dependencies from #15172, #15261, #15263, #15302 to #15172, #15261, #15263, #15370 #15302

[vbraun]

Description changed:

--- 
+++ 
@@ -10,7 +10,7 @@
 * fricas-aldor (dito)
 * libcocoa (doesn't build with our gmp/mpir)

-Delete because no copyright information (ask again on sage-devel first):
+Delete because no SPKG.txt:
 * cunningham_tables
 * database_jones_numfield
 * database_kohel

[vbraun]

Description changed:

--- 
+++ 
@@ -5,12 +5,7 @@
 The optional databases will be dealt with here, other packages should make separate tickets. Use the following new "upstream" tarballs:
 * http://boxen.math.washington.edu/home/vbraun/upstream/database_cremona_ellcurve-20121022.tar.bz2

-Move to experimental (broken with or without git, probably for a long time):
-* fricas (doesn't build with our ecl)
-* fricas-aldor (dito)
-* libcocoa (doesn't build with our gmp/mpir)
-
-Delete because no SPKG.txt:
+Leave for later, don't work and no SPKG.txt:
 * cunningham_tables
 * database_jones_numfield
 * database_kohel
@@ -18,6 +13,11 @@
 * database_sloane_oeis
 * database_stein_watkins_mini.p0

+Move to experimental (broken with or without git, probably for a long time):
+* fricas (doesn't build with our ecl)
+* fricas-aldor (dito)
+* libcocoa (doesn't build with our gmp/mpir)
+
 Delete for other reasons:
 * jsmath_image_fonts (we switched to mathjax)

[vbraun]

comment:26

I'm setting this to positive review since clearly nobody else has the guts to review this ticket.

The database_cremona_ellcurve-20121022.tar.bz2 tarball is already in the upstream dir, apparently.

Harald, can you delete/move to trash: fricas, fricas-aldor, libcocoa, and jsmath_image_fonts? They are (and have been for a while) useless with or without git.

[vbraun]

Reviewer: Volker Braun

[haraldschilly]

comment:29

Replying to @vbraun:

Harald, can you delete/move to trash: fricas, fricas-aldor, libcocoa, and jsmath_image_fonts?

ok.

[JohnCremona]

comment:30

My copy of database_stein_watkins_mini.p0.spkg does contain SPKG.txt:

jec@atkin:~/spkg/database_stein_watkins_mini.p0$ ls -l
total 12
-rwx------ 1 jec jec   49 Mar 12  2010 spkg-install
-rw------- 1 jec jec  389 Mar 13  2010 SPKG.txt
drwx------ 2 jec jec 4096 Nov  7  2005 stein-watkins-ecdb

so I think this one should be easy -- no compilaton, it just copies recurseively the directory to SAGE_DATA:

jec@atkin:~/spkg/database_stein_watkins_mini.p0$ cat spkg-install 
#!/bin/sh

cp -rf stein-watkins-ecdb $SAGE_DATA/

[JohnCremona]

comment:31

If I was going to work on converting stein_watkins_mini, should I open a new ticket to hold the branch?

[vbraun]

comment:32

Please open a new ticket, this one is closed.

stein_watkins_mini contains a skeleton SPKG.txt, but no description (what is it anyways?) and no license (=potentially not legal to distribute). William indicated in the ensuing thread on sage-devel that it is public domain afair.

[JohnCremona]

comment:33

Replying to @vbraun:

Please open a new ticket, this one is closed.

See #15556.

stein_watkins_mini contains a skeleton SPKG.txt, but no description (what is it anyways?) and no license (=potentially not legal to distribute). William indicated in the ensuing thread on sage-devel that it is public domain afair.

It's a large collection of elliptic curves compiled by William Stein and Mark Watkins about 10 years ago, stored in simple compressed text files. I'll put a reference to their paper in the SPKG.txt (it is already cited properly in the docstrings).

[rwst]

comment:34

See also #15814

[rwst]

Changed dependencies from #15172, #15261, #15263, #15370 #15302 to #15172, #15261, #15263, #15370 #15302, #15814