command line option parsing

[dimpase]

Issue created by migration from Trac.

Original creator: @williamstein

Original creation time: 2006-09-12 23:21:18

Assignee: somebody

CC: kini saraedum iandrus

We should improve and/or modernize and/or revise Sage's command-line parsing.

Two ideas, which could be debated endlessly and could also be implemented independently of each other:

• use Python's argparse module to handle the parsing, rather than a shell script
• change Sage's options from flags to subcommands: sage --package ... would be changed to sage package ..., etc. (comment:56 and comment:57 list some possibilities)

(changed by jhpalmieri at 2020-08-11 17:35:51)

[dimpase]

Comment by @williamstein created at 2007-01-13 01:56:18

no -- you can't combine command line options like that. this isn't a bug but a not implemented yet issue.

[dimpase]

Comment by @williamstein created at 2007-01-13 01:56:18

Changing type from defect to enhancement.

[dimpase]

Comment by mabshoff created at 2007-09-11 02:11:39

This should be fixable, but the long term goal is to do a proper rewrite of the command line options.

Cheers,

Michael

[dimpase]

Comment by gfurnish created at 2008-03-16 07:59:18

Changing assignee from somebody to gfurnish.

[dimpase]

Comment by gfurnish created at 2008-03-16 07:59:18

Changing status from new to assigned.

[dimpase]

Comment by gfurnish created at 2008-04-04 19:55:55

Changing component from basic arithmetic to interfaces.

[dimpase]

Comment by mabshoff created at 2008-09-24 02:59:36

See also #180 for a bunch of related failures due to the option parsing being dumb :o

Cheers,

Michael

[dimpase]

Comment by mabshoff created at 2008-09-24 02:59:36

Changing status from assigned to new.

[dimpase]

Comment by mabshoff created at 2008-09-24 02:59:36

Changing assignee from gfurnish to mabshoff.

[dimpase]

Comment by mabshoff created at 2008-09-24 02:59:46

Changing status from new to assigned.

[dimpase]

Comment by kcrisman created at 2009-12-30 05:15:23

Note that sage -bn now builds, then does notebook, though of course it doesn't fix the issue here.

[dimpase]

Comment by jhpalmieri created at 2010-03-19 15:41:48

the file SAGE_ROOT/makefile

[dimpase]

Attachment makefile by jhpalmieri created at 2010-03-19 15:41:55

[dimpase]

Attachment makefile.diff by jhpalmieri created at 2010-03-19 15:42:11

extcode repo

[dimpase]

Attachment trac_21-extcode.patch by jhpalmieri created at 2010-03-19 15:42:32

sagenb repo

[dimpase]

Attachment trac_21-sagenb.patch by jhpalmieri created at 2010-03-19 15:50:37

Here are patches. After applying "trac_21-scripts.patch", you may need to make "SAGE_ROOT/local/bin/sage-sage.py" executable. The build process works for me with these patches. For the standard packages, the third line in

if [ "$SAGE_LOCAL" = "" ]; then
   echo "SAGE_LOCAL undefined ... exiting";
   echo "Maybe run 'sage -sh'?"
   exit 1
fi

should be changed to "Maybe run 'sage --sh'?", but this doesn't affect the functioning of the packages, and otherwise, they don't need changing. I haven't looked at optional packages.

This approaches uses Python's optparse to parse command-line options. If someone wants to write a version using shflags or some other package, go ahead.

I propose the following approach, whether using these patches or other ones:

• first, we include new command-line options but don't turn them on by default, instead printing a message like this one when you type "sage [...]" with a nonempty argument:

  Note: Using old-style Sage command-line options. 
  
  To try out Sage's experimental GNU/Posix-style command-line options 
  (for example, 'sage --notebook' instead of 'sage -notebook'), set the 
  environment variable $SAGE_NEW_OPTIONS to something nonempty. 
  To bypass this message, set the environment variable 
  $SAGE_SKIP_OPTIONS_MESSAGE to something nonempty. 

  Running "sage" (with no arguments) would not trigger this message. (Perhaps we could only turn this on in prerelease (alpha and rc) versions? Alternatively, a change like this could go with the version 5.0 release.)

• after a while, we switch this to

  Warning: Using old-style Sage command-line options. 
  
  Sage is changing to use conventional GNU/Posix-style command-line options 
  (for example, 'sage --notebook instead of 'sage -notebook).  This change will 
  become the default soon.  Meanwhile, to use this new style (and therefore 
  to avoid seeing this message), set the environment variable 
  $SAGE_NEW_OPTIONS to something nonempty. 

  perhaps with no easy way of disabling this message while using old-style options.

• finally, we turn on the new options, perhaps with an environment variable $SAGE_OLD_OPTIONS to use the old ones, with the understanding that any changes in command-line options may not be maintained for the old version.

See sage-devel for some discussion.

[dimpase]

Comment by jhpalmieri created at 2010-03-19 15:50:37

Changing status from new to needs_review.

[dimpase]

Comment by jhpalmieri created at 2010-03-19 15:50:47

Changing priority from minor to critical.

[dimpase]

Comment by jhpalmieri created at 2010-03-19 20:25:24

I've marked this as "needs review", but it might need work. In the previously cited thread from sage-devel, there was the following suggestion:

Another possibility might be to first check for "--gp", "--gap", etc., 
and do those before doing the general option parsing.   I.e., just do 
what you already planned, but with one optimization to deal with this 
use case. 

This is to speed up access to these programs: do a check like this in a shell script, and then pass the rest of the arguments to Python's optparse using the script included in this patch, or one like it. Then you avoid the slight delay involved in starting up Python if you want to run "gp". It would be nice to have a shell script which had a list of strings "gp", "gap", etc., checked to see if the first(?) argument was "--STR" for STR in this list, and if so, run the appropriate program from SAGE_LOCAL/lib, passing the rest of the line as arguments. Having one list containing all of these strings would make it easy to customize.

[dimpase]

Attachment sage by jhpalmieri created at 2010-03-19 21:15:39

the file SAGE_ROOT/sage

[dimpase]

Attachment sage.diff by jhpalmieri created at 2010-03-19 21:22:45

Replying to [comment:10 jhpalmieri]:

I've marked this as "needs review", but it might need work. In the previously cited thread from sage-devel, there was the following suggestion:

Another possibility might be to first check for "--gp", "--gap", etc., 
and do those before doing the general option parsing.   I.e., just do 
what you already planned, but with one optimization to deal with this 
use case. 

Okay, here's a new version which does this: it adds a file sage-sage-quickstart which gets run first, implementing the above idea. Then if SAGE_NEW_OPTIONS is nonempty, it calls sage-sage.py, the Python/optparse version with GNU/Posix standard command-line options. Otherwise, it calls the old parser sage-sage.

For the record, the commands in sage-sage-quickstart are: axiom, ecl/lisp, gap, gp, hg, ipython, maxima, mwrank, python, R, singular. Are any others particularly sensitive to startup times? (Using python adds something less than .1 second on my two-year old iMac, so we're not talking about a lot of time, in any case.)

[dimpase]

Comment by jhpalmieri created at 2010-05-07 19:06:03

Note that Python 2.7 will include the argparse module, which might be easier to use than optparse.

[dimpase]

Comment by @williamstein created at 2010-06-03 04:33:02

Changing status from needs_review to positive_review.

[dimpase]

Comment by @williamstein created at 2010-06-03 04:33:02

Wow, this is really fantastic.

[dimpase]

Comment by @williamstein created at 2010-06-03 04:36:19

Changing status from positive_review to needs_work.

[dimpase]

Comment by @williamstein created at 2010-06-03 04:37:18

Changing status from needs_work to needs_review.

[dimpase]

Comment by @williamstein created at 2010-06-03 04:37:18

I'm changing this back to needs review. I realized when trying to apply it that I had got confused about how to even apply this.

John, can you please post clear directions about how to use the patches attached to this ticket?

sage: hg_scripts.apply('http://trac.sagemath.org/sage_trac/attachment/ticket/21/trac_21-scripts.patch')
Attempting to load remote file: http://trac.sagemath.org/sage_trac/raw-attachment/ticket/21/trac_21-scripts.patch
Loading: [..........]
cd "/mnt/usb1/scratch/wstein/build/release/4.4.3/sage-4.4.3.alpha2/local/bin" && hg status
cd "/mnt/usb1/scratch/wstein/build/release/4.4.3/sage-4.4.3.alpha2/local/bin" && hg status
cd "/mnt/usb1/scratch/wstein/build/release/4.4.3/sage-4.4.3.alpha2/local/bin" && hg import   "/scratch/wstein/sage/temp/sage.math.washington.edu/22653/tmp_0.patch"
applying /scratch/wstein/sage/temp/sage.math.washington.edu/22653/tmp_0.patch
patching file sage-pkg
Hunk #1 FAILED at 40
Hunk #2 FAILED at 63
2 out of 2 hunks FAILED -- saving rejects to file sage-pkg.rej
patching file sage-run
Hunk #1 FAILED at 17
1 out of 1 hunks FAILED -- saving rejects to file sage-run.rej
patching file sage-sage
Hunk #2 FAILED at 34
Hunk #3 FAILED at 196
Hunk #4 FAILED at 212
Hunk #5 FAILED at 424
Hunk #6 FAILED at 450
Hunk #8 FAILED at 608
Hunk #9 FAILED at 744
Hunk #10 FAILED at 767
Hunk #11 succeeded at 894 with fuzz 1 (offset 0 lines).
8 out of 11 hunks FAILED -- saving rejects to file sage-sage.rej
abort: patch failed to apply

[dimpase]

Comment by jhpalmieri created at 2010-06-03 06:02:38

scripts repo

[dimpase]

Attachment trac_21-scripts.patch by jhpalmieri created at 2010-06-03 06:02:54

sage repo

[dimpase]

Attachment trac_21-sage.patch by jhpalmieri created at 2010-06-03 06:04:06

John, can you please post clear directions about how to use the patches attached to this ticket?

Sorry, some parts needed rebasing. I think it's okay now. I've modified the summary with instructions for how to apply the patches.

[dimpase]

Comment by ohanar created at 2010-11-11 19:54:30

It appears like sage-sage.py managed to get itself into 4.4.3 (William, could this have happened when you were trying to apply it). Also, sage-apply-ticket has greatly changed since the patches were posted, so I am changing this back to needs work.

[dimpase]

Comment by ohanar created at 2010-11-11 19:54:30

Changing status from needs_review to needs_work.

[dimpase]

Comment by jhpalmieri created at 2010-11-11 20:08:59

As far as I know, the file sage-sage.py was accidentally added in 4.4.3, but the release manager should probably delete it: it's not used anywhere.

I don't have the time to rebase it right now, so if anyone else wants to work it, that would be great.

[dimpase]

Attachment 21-scripts-4.6.1.alpha1.patch by ohanar created at 2010-11-12 10:48:35

scripts repo, rebased on 4.6.1.alpha1

[dimpase]

Comment by ohanar created at 2010-11-12 11:01:28

I've uploaded a patch rebased on 4.6.1.alpha1 (I also added in #8654 while I was at it). One thing I noted during rebasing it is that -tp no longer works (and hence -btp which was added since 4.4.3), I don't know optparse well enough to come up with a solution (if we want to provide one). The documentation patches will need to be rebased as well.

For anyone who wants to apply this, make sure to remove sage-sage.py before applying, the file is currently just hanging around not doing anything - mercurial doesn't even know about it.

[dimpase]

Comment by kini created at 2011-11-01 14:49:12

ohanar mentioned on #8654 that this should be rewritten (or modified) to use argparse as optparse is deprecated in Python 2.7, so I'm adding #9958 to the dependencies because argparse is shipped with 2.7. Alternatively we could make an spkg for argparse.

[dimpase]

Comment by jhpalmieri created at 2011-11-02 01:27:23

I think waiting until 2.7 is fine.

[dimpase]

Comment by kini created at 2011-11-13 10:39:14

Can we edit the title and description of this ticket to reflect what we're actually trying to do here? (I would, but preferably someone who has been working on it should do it.) The goal is to completely replace the command line handler (currently a shell script) with a Python script which uses argparse for extensibility, right?

[dimpase]

Comment by kini created at 2012-01-19 12:14:37

People CC'd to this ticket probably already know this, but the above mentioned #9958 is merged, so we can use Python 2.7 default modules such as argparse now.

[dimpase]

Comment by ohanar created at 2012-01-24 17:20:32

Replying to [comment:26 kini]:

People CC'd to this ticket probably already know this, but the above mentioned #9958 is merged, so we can use Python 2.7 default modules such as argparse now.

Yup, I've started working on this, but its going to take awhile... :)

[dimpase]

Comment by kini created at 2012-01-24 17:34:26

Capital! :) As it's a large project maybe you could post WIPs once in a while?

I was planning on diving into the scripts dir myself and trying to work on this, but I guess it should be an order of magnitude easier for you, haha. Still, let me know if I can help with anything.

[dimpase]

Comment by ohanar created at 2012-01-26 10:15:07

Replying to [comment:28 kini]:

Capital! :) As it's a large project maybe you could post WIPs once in a while?

No problem, once I have something of any real substance, I'll be sure to post it somewhere.

I was planning on diving into the scripts dir myself and trying to work on this, but I guess it should be an order of magnitude easier for you, haha. Still, let me know if I can help with anything.

Well, I'm just starting and have had a busy week with other stuff, so you would probably be in about as good of position as myself, especially since we need to rebase this off of the 5.0 dev builds, which changed a lot of that stuff anyway.

I'm finding myself annoyed at spkg/script, and all the little special cases that we have. I really just want to rip that out and try to have a (more) unified design to our parsing. My current fancy is to introduce a bunch of subcommands, like mercurial or aptitude. Some of the ones I've thought about:

% sage ARGS # this would be for running sage scripts, or a couple of oddball arguments
% sage notebook ARGS
% sage pkg ARGS # this would include spkg stuff
% sage pkg install # since install has some special flags like -f or -s
% sage test ARGS
% sage build ARGS
% sage {python,sqlite3,R,gp,...} ARGS # we can consider these programs as subcommands of sage

I haven't fully worked out what this would look like with all the arguments (such as debugging), but IMO it would greatly clean up our command line tools. Also, this would simplify many aspects of the implementation, although some hacking of argparse will still have to be done (it currently doesn't support optional subparsers, see [http://bugs.python.org/issue9253]).

I probably should bring this up on the devel list, but I'm tired and should go to bed before I have to be up in the morning :/.

[dimpase]

Comment by kini created at 2012-05-25 20:16:23

We're going to try to get this in as a Sage 6.0 goal along with #13015.

[dimpase]

Comment by kini created at 2012-05-25 23:06:21

patchbot: apply 21-scripts-4.6.1.alpha1.patch

(this doesn't actually work but at least this way the patchbot won't try to apply the other stuff, and even crash, apparently)

[dimpase]

Comment by jdemeyer created at 2013-01-24 13:30:48

Going with argparse is probably a good idea, but I would still like a special "pre-parser" in bash to handle to options

sage --sh
sage -i
sage -f

(or whatever version they will become)

I think these must be available from the start, before Python has been built.

[dimpase]

Comment by jdemeyer created at 2013-01-24 13:30:48

Changing assignee from mabshoff to jdemeyer.

[dimpase]

Comment by kini created at 2013-01-24 17:39:13

ohanar and I talked about this a bit at SD40.5. What do you think about having a totally separate executable (rather than a preparser) called sage-sh? Stuff like the current sage -gap could be called rapidly with sage-sh -c gap, or slowly via python with sage gap or what-have-you.

[dimpase]

Comment by jhpalmieri created at 2013-01-24 18:33:18

William, among others, as alluded to [comment:11 above], was pretty insistent that commands like sage --gp execute quickly, without the overhead that starting Python entails. Hence the approach in my patches on this ticket. If you want to insist that people use sage-sh -c gp instead, I don't think it can get a positive review unless the approach gets a positive response on sage-devel.

[dimpase]

Comment by kini created at 2013-01-24 18:44:15

Rather than building on your patches on this ticket, we're proposing completely rewriting the command line interface of Sage, and that includes changing invocation methods. That's why we've set the milestone to sage-6.0 - the idea is to tie it with the layout restructuring and new development interface that will come with the git transition (#13015). We figured that the git transition is a big enough change to warrant a major version bump (though that might end up being sage-7.0 or whatever, depending on the timeline).

So, of course, this will all go through sage-devel in time, once we figure out exactly what we're proposing.

[dimpase]

Comment by jhpalmieri created at 2013-01-24 18:57:09

Oh, I understood that you were starting from scratch. I was responding to your proposal about sage-sh, suggesting that not all users would be happy with that solution.