search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.38k stars 471 forks source link

Move SAGE_ROOT/build/pkgs/*/src to SAGE_ROOT/pkgs/* #31577

Closed mkoeppe closed 3 years ago

mkoeppe commented 3 years ago

In the current layout, these embedded source trees are a little difficult to find / complicated to type.

The proposed location is:

SAGE_ROOT
- pkgs
  - sage-conf
    - setup.py
    - bin/
    - sage_conf.py.in
  - sage-docbuild
    - setup.py
    - sage_docbuild   -> symlinks to SAGE_ROOT/sage_docbuild/
  - sage-sws2rst
    - setup.py
    - bin/
    - sage_sws2rst/
  - sagemath-standard
    - setup.py
    - bin             -> symlinks to SAGE_ROOT/src/bin/        
    - sage            -> symlinks to SAGE_ROOT/src/sage/
    - sage_setup      -> symlinks to SAGE_ROOT/src/sage_setup/ (removed in #29847)

that is, the new directory pkgs will be on the same level as 

SAGE_ROOT
- src

the unchanged monolithic sagelib source tree, which will continue to contain subdirectories

  - bin/ 
  - doc/
  - sage/
  - sage_docbuild/      # was sage_setup/docbuild/ in Sage 9.2
  - sage_setup/

We add symlinks from the previous locations SAGE_ROOT/build/pkgs/sage[math]_*/src to the new locations SAGE_ROOT/pkgs/sage[math]-*

29847 will add:

SAGE_ROOT
- pkgs
  - sage-setup
    - setup.py
    - sage_setup         -> symlinks to src/sage_setup/

and remove sage_setup from the sagemath-standard distribution.

Other modularization tickets (#29705) will add

SAGE_ROOT
- pkgs
  - sagemath-core/
  - sagemath-brial/
  - sagemath-giac/
  - sagemath-meataxe/
  - sagemath-tdlib/

etc.

29868 will add

SAGE_ROOT
- pkgs
  - sagemath-doc-html/
  - sagemath-doc-pdf/

All files that contain Sage doctests will remain in the monolithic src/ source tree; the source trees of the distributions symlink there. This may be changed in a follow-up ticket regarding the modularization of doctesting.

Before merging this branch into a non-distclean worktree, it is useful to use 

  git clean -f -x build/pkgs/*/src

CC: @jhpalmieri @dimpase @kiwifb

Component: build

Author: Matthias Koeppe

Branch/Commit: 7568dc6

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/31577

jhpalmieri commented 3 years ago
comment:39

Replying to @kiwifb:

You are right, it will skipped. Although technically they could be run if present.

One of the goals of that particular method is to exclude files if in a distro. I don't know enough about distros to know if the method is well-implemented, though. (I didn't write it.)

Some related questions about this ticket: how does this new organization affect distros, and should we be doctesting any of the pieces that are being moved to SAGE_ROOT?

mkoeppe commented 3 years ago
comment:40

Replying to @jhpalmieri:

should we be doctesting any of the pieces that are being moved to SAGE_ROOT?

sage-sws2rst has its own simple test script, not related to the Sage doctester.

The other packages do not have separate tests but are doctested as part of the Sage. The assumption is here that the doctester is run after all of these packages are installed. This assumption would continue to hold even as the modularization goes forward. I don't really have a plan for modularizing the doctesting, other than #30778.

mkoeppe commented 3 years ago
comment:41

Replying to @jhpalmieri:

how does this new organization affect distros

I would hope, not at all. All paths that were valid before are still valid afterwards thanks to the symlinks build/pkgs/SOME_SPKG/src -> /SOME-SPKG

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -35,6 +35,8 @@
     - sage_docbuild/      # was sage_setup/docbuild/ in Sage 9.2
     - sage_setup/

+ +We add symlinks from the previous locations SAGE_ROOT/build/pkgs/sage[math]_*/src to the new locations SAGE_ROOT/sage[math]-*

In follow-up ticket #31662, we rename the confusingly named directory SAGE_ROOT/build (Python users expect a directory named build to contains build artifacts only) to

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -56,6 +56,7 @@
     - setup.py
     - sage_setup         -> symlinks to src/sage_setup/

+and remove sage_setup from the sagemath-standard distribution.

Other modularization tickets (#29705) will add

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -19,7 +19,7 @@
     - setup.py
     - bin             -> symlinks to SAGE_ROOT/src/bin/        
     - sage            -> symlinks to SAGE_ROOT/src/sage/
-    - sage_setup      -> symlinks to SAGE_ROOT/src/sage_setup/
+    - sage_setup      -> symlinks to SAGE_ROOT/src/sage_setup/ (removed in #29847)

that is, these new directories will be on the same level as

kiwifb commented 3 years ago
comment:45

Replying to @mkoeppe:

Replying to @jhpalmieri:

how does this new organization affect distros

I would hope, not at all. All paths that were valid before are still valid afterwards thanks to the symlinks build/pkgs/SOME_SPKG/src -> /SOME-SPKG

As far as I am concerned, anything living under build/pkgs/ should have a separate tarball and be installed as a separate package. This has been somewhat challenged by the way Matthias is conducting the modularization. But that's the goal at the end of the day. If it is there it is not supposed to be anything special anymore.

That being said sage_conf is being a pain because it is still very special as far as I can see. It still need a real SAGE_ROOT tree. But that is a different conversation and ticket.

mkoeppe commented 3 years ago
comment:46

That's right, after this ticket, SAGE_ROOT/sage-docbuild, SAGE_ROOT/sage-sws2rst, SAGE_ROOT/sagemath-standard, and SAGE_ROOT/sage-conf are standard source trees of Python distribution packages, containing nothing that is specific to Sage-the-distribution.

It is possible to build sdists from each of these source trees using setup.py sdist, and wheels using setup.py bdist_wheel.

Actually also sage-conf has the same status: You can build an sdist or a wheel from it. However, this version of sage-conf is not useful for putting it on PyPI. #29039 and #31396 create separate versions of sage-conf that are suitable for distribution.

kiwifb commented 3 years ago
comment:47

Replying to @mkoeppe:

Actually also sage-conf has the same status: You can build an sdist or a wheel from it. However, this version of sage-conf is not useful for putting it on PyPI. #29039 and #31396 create separate versions of sage-conf that are suitable for distribution.

That's right, I had to remove a lot of stuff from the PyPI version before I could make it do the bits that would be useful to me properly as someone who build stuff from source. Considering the amount of stuff I just want to throw out, it is almost a different package altogether.

mkoeppe commented 3 years ago
comment:48

Yes, downstream packaging is encouraged to write a separate sage_conf for their purposes.

kiwifb commented 3 years ago
comment:49

Replying to @mkoeppe:

Yes, downstream packaging is encouraged to write a separate sage_conf for their purposes.

Beurk. Anyway, we should have another meeting may be the next sage-days. I think there are things we should do to the design of "those" configuration files at large, and that could make it easier for distros in the process. I need to figure if what I think of who be easily doable python side. And it is definitely a separate conversation from this ticket.

mkoeppe commented 3 years ago
comment:50

It's really very easy:

kiwifb commented 3 years ago
comment:51

Replying to @mkoeppe:

It's really very easy:

  • You don't want to run configure -> write your sage_conf.py in which you hardcode your stuff.

This is probably the best solution and it is also incredibly dirty once you get QA and multi-python support involved. In short this is cheap and easy if your packaging is somewhat ad hoc. Once you want this to go inside a framework with rules, there is a higher price than you may expect.

But we are getting incredibly off-topic and should stop polluting this ticket.

mkoeppe commented 3 years ago
comment:52

Yes, let's continue with this at another time and on another ticket.

jhpalmieri commented 3 years ago
comment:53
% ./sage -t src/sage/misc/package.py 
Running doctests with ID 2021-04-12-22-39-35-af3c76ef.
Git branch: t/31577/move_sage_root_build_pkgs___src_to_sage_root____rename_sage_root_build_to_sage_root_sage_bootstrap
Using --optional=build,dochtml,homebrew,pip,sage,sage_spkg
Doctesting 1 file.
sage -t --warn-long 125.4 --random-seed=0 src/sage/misc/package.py
**********************************************************************
File "src/sage/misc/package.py", line 336, in sage.misc.package.installed_packages
Failed example:
    installed_packages()['sage_conf']  # optional - build
Expected:
    'none'
Got:
    '9.3.rc3'
**********************************************************************
1 item had failures:
   1 of   4 in sage.misc.package.installed_packages
    [50 tests, 1 failure, 2.86 s]
----------------------------------------------------------------------
sage -t --warn-long 125.4 --random-seed=0 src/sage/misc/package.py  # 1 doctest failed
----------------------------------------------------------------------
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 08b8306 to 248fae7

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

248fae7src/sage/misc/package.py: sage_conf now has a version
mkoeppe commented 3 years ago
comment:55

Thanks for testing!

mkoeppe commented 3 years ago

Changed dependencies from #30913 to #30913, #31427

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 248fae7 to 9dac5c7

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

32c069a.gitignore: Integrate src/{doc,sage}/.gitignore here so that Docker ignores the listed files too
9dac5c7Merge #31427
mkoeppe commented 3 years ago

Work Issues: Fix up "tox -e docker..."

jhpalmieri commented 3 years ago
comment:59

What are the files Pipfile, Pipfile.m4, etc. for? I just build Sage, and for example, sagemath-standard/Pipfile is currently a link pointing to a nonexistent target, and the same for sagemath-standard/Pipfile-dist.

mkoeppe commented 3 years ago
comment:60

If you run ./bootstrap, the targets of these symlinks should be generated.

These files were introduced in #30913. They are not used in a normal build. They are for users who want to use https://pypi.org/project/pipenv/ to set up a virtual environment.

jhpalmieri commented 3 years ago
comment:61

I might prefer this in place of, or in addition to, the fixed installed_packages() doctest:

diff --git a/src/sage/misc/package.py b/src/sage/misc/package.py
index e402293baf..f9a3f9fc9e 100644
--- a/src/sage/misc/package.py
+++ b/src/sage/misc/package.py
@@ -335,6 +335,8 @@ def installed_packages(exclude_pip=True):
         '0.7.12'
         sage: installed_packages()['sage_conf']  # optional - build, random
         '9.3'
+        sage: installed_packages()['sage_conf'] == sage.env.SAGE_VERSION # optional - build
+        True

     .. SEEALSO::
mkoeppe commented 3 years ago
comment:62

That's a good idea

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 9dac5c7 to 58bee8e

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

58bee8esrc/sage/misc/package.py: Better doctest for installed_packages
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 58bee8e to 16b54d1

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

16b54d1build/pkgs/{sage_docbuild,sage_sws2rst}/package-version.txt: Symlink to ../sagelib/package-version.txt; do not symlink into src to make things easier for the Docker build
mkoeppe commented 3 years ago
comment:65

Unfortunately I have a bit of trouble to make this layout work well with the tox-docker build.

How about pushing the new directories down one level, to something like SAGE_ROOT/pkgs/sage-conf etc.?

jhpalmieri commented 3 years ago
comment:66

Should the Sage source be in the same place? Or rather, should everything be in src? I don't know if there is any standard way to do this or any particular logic behind it.

mkoeppe commented 3 years ago
comment:67

I don't think there is a standard for this.

I initially had made it SAGE_ROOT/src/pkgs/sage-conf but I think it would be cleaner if src only contained the monolithic Sage library.

jhpalmieri commented 3 years ago
comment:68

Then sure, SAGE_ROOT/pkgs/sage-conf sounds good.

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -4,6 +4,7 @@

SAGE_ROOT +- pkgs

mkoeppe commented 3 years ago

Changed work issues from Fix up "tox -e docker..." to Make it SAGE_ROOT/pkgs/...

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -42,6 +42,8 @@
 #29847 will add:

+SAGE_ROOT +- pkgs

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -25,6 +25,7 @@
 that is, the new directory `pkgs` will be on the same level as

+SAGE_ROOT

mkoeppe commented 3 years ago

Changed work issues from Make it SAGE_ROOT/pkgs/... to Make it SAGE_ROOT/pkgs/..., merge #31825

mkoeppe commented 3 years ago

Changed dependencies from #30913, #31427 to #30913, #31427, #31825

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

9c3b887Move /build/pkgs/sage_docbuild/src to /sage_docbuild, leave symlink behind
1628441Move /build/pkgs/sage_sws2rst/src to /sage_sws2rst, leave symlink behind
a5d164bMove /build/pkgs/sagelib/src to /sagemath_standard, leave symlink behind
673de88.gitignore: Update for new source tree locations
2bdd5bdSpell distribution source directories sage-conf, sage-docbuild, sage-sws2rst, sagemath-standard with dashes, not underscores
e35a6d1Also standardize on using dashes in distribution names in setup.cfg
89c78e8WIP: Fix up docker build
fac294dbuild/pkgs/{sage_docbuild,sage_sws2rst}/package-version.txt: Symlink to ../sagelib/package-version.txt; do not symlink into src to make things easier for the Docker build
f0f5d54build/bin/write-dockerfile.sh: Adjust ADD commands
6ad0b8fWIP update .gitignore/.dockerignore
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 16b54d1 to 6ad0b8f

mkoeppe commented 3 years ago

Changed dependencies from #30913, #31427, #31825 to none

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

c58e51bMove package source directories to pkgs/
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 6ad0b8f to c58e51b

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from c58e51b to 1d23923

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

ce3c4d3Move /build/pkgs/sage_conf/src to /sage_conf, leave symlink behind
7824e0bAlso standardize on using dashes in distribution names in setup.cfg
49323d2build/pkgs/{sage_docbuild,sage_sws2rst}/package-version.txt: Symlink to ../sagelib/package-version.txt; do not symlink into src to make things easier for the Docker build
1d23923update .gitignore/.dockerignore; Move package source directories to pkgs/
mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -38,7 +38,7 @@
   - sage_setup/

-We add symlinks from the previous locations SAGE_ROOT/build/pkgs/sage[math]_*/src to the new locations SAGE_ROOT/sage[math]-* +We add symlinks from the previous locations SAGE_ROOT/build/pkgs/sage[math]_*/src to the new locations SAGE_ROOT/pkgs/sage[math]-*

29847 will add:

@@ -73,7 +73,7 @@

-All files that contain Sage doctests will remain in the monolithic src/ source tree; the source trees of the distributions symlink there. This may be changed in a follow-up ticket regarding the modularization of doctesting. +All files that contain Sage doctests will remain in the monolithic src/ source tree; the source trees of the distributions symlink there. This may be changed in a follow-up ticket regarding the modularization of doctesting.

mkoeppe commented 3 years ago

Changed work issues from Make it SAGE_ROOT/pkgs/..., merge #31825 to none

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

0cda8f1README: Update directory chart