Clarify and complete documentation of function()

[94a1bc6f-6005-414c-9598-64422d48cc95]

The documentation of function() is incomplete and confusing. For example, none of the methods described in http://www.sagemath.org/doc/reference/calculus/sage/symbolic/function_factory.html show up when I type:

function?

The distinction between

f = function('f')

and

f = function('f', x)

is also not documented. See also #17445 for sources of confusion. Need to explain what happens in the second case above (f = function('f',x)). A symbolic function f is first created and then overwritten by the expression f? See the following example, where f is first an expression, then becomes redefined to a function in the background, but does not contain any information about its variables.

sage: f = function('f', x); print type(f)
<type 'sage.symbolic.expression.Expression'>
sage: fx = function('f',x); print type(f)
<class 'sage.symbolic.function_factory.NewSymbolicFunction'>
sage: f.variables()
()
sage: fx.variables()
(x,)

See http://ask.sagemath.org/question/9932/how-to-substitute-a-function-within-derivatives/?answer=14752#post-id-14752 for a possible start at how to explain this, at least for those writing this.

CC: @nbruin @kcrisman @sagetrac-tmonteil @zimmermann6

Component: documentation

Author: Nils Bruin, Ralf Stephan

Branch/Commit: 7c029f3

Reviewer: Ralf Stephan, Nils Bruin

Issue created by migration from https://trac.sagemath.org/ticket/17447

[kcrisman]

Description changed:

--- 
+++ 
@@ -14,7 +14,7 @@

f = function('f', x)

-is also not documented. See also https://github.com/sagemath/sage-prod/issues/17445 for sources of confusion. Need to explain what happens in the second case above (`f = function('f',x)`). A symbolic function `f` is first created and then overwritten by the expression `f`?
+is also not documented. See also #17445 for sources of confusion. Need to explain what happens in the second case above (`f = function('f',x)`). A symbolic function `f` is first created and then overwritten by the expression `f`?
 See the following example, where f is first an expression, then becomes redefined to a function in the background, but does not contain any information about its variables. 

[kcrisman]

Description changed:

--- 
+++ 
@@ -29,3 +29,4 @@

+See http://ask.sagemath.org/question/9932/how-to-substitute-a-function-within-derivatives/?answer=14752#post-id-14752 for a possible start at how to explain this, at least for those writing this.

[nbruin]

comment:3

For the question about creation and overwriting:

The toplevel function has the sideeffect of inserting a binding in the global scope, just as the toplevel var does.

sage: var('U')
U
sage: U
U
sage: function('f')
f
sage: f
f

The library versions of these routines don't do that:

sage: SR.var('V')
V
sage: V
NameError: name 'V' is not defined
sage: sage.symbolic.function_factory.function('g')
g
sage: g
NameError: name 'g' is not defined

The intended use of the toplevel function is not to be assigned but to be called as a statement, just as var. There is plenty of documentation in sage that was written by people unaware of this fact (or possibly they think they're being helpful by providing code that doesn't return an error when used with the library versions of these functions)

As a result x=var('x') and fx=function('f',x) are quite ubiquitous in the docs. The former is fairly inocuous but the second, as you experience, has a rather nasty side effect. We're violating python's convention here: routines with side effects should return None.

I think this is the point where people usually give up fixing this mess (I have). Hopefully you will now follow through.

Note that we can probably not do more than document the issues, which is not going to fix much, because people won't read the doc, but then at least you can point them somewhere when they complain.

I suspect that Zimmerman's calculus book uses this stuff quite extensively, so deprecating/changing the current behaviour will likely lead to pushback from him, and probably rightly so.

(this is the kind of issue that earns sage the reputation of "implement now, design later")

[94a1bc6f-6005-414c-9598-64422d48cc95]

comment:5

Awesome, thank you, Nils. These things always confused me, so I'm relieved to hear that they should be avoided! Even the documentation of var is full of such examples:

sage: x = var('x', domain=RR); x; x.conjugate()
x
x
sage: y = var('y', domain='real'); y.conjugate()
y
sage: y = var('y', domain='positive'); y.abs()
y

Custom latex expression can be assigned to variable:

sage: x = var('sui', latex_name="s_{u,i}"); x._latex_()
's_{u,i}'

Should we change these examples, too?

[nbruin]

Branch: u/nbruin/clarify_and_complete_documentation_of_function__

[nbruin]

comment:6

Don't base any other work on this branch just yet, since it's likely to be rebased. I have deprecated the use of function('f',x) in favour of using function('f')(x). It's hardly longer to type and much less ambiguous. Otherwise, it seems that the documentation here is actually pretty accurate, apart from being terse. As soon as the objects returned by function('f') are consistent, I think users will be forced to learn the (now clear) semantics pretty quickly.

The object returned by function(f) doesn't allow symbolic operations on it, so people will quickly find they need to evaluate it in order to get a symbolic expression.

I haven't changed all doctests to take into account the deprecation, but I've done some files to show the impact of the change (seems relatively minor).

[nbruin]

Commit: fa3ebd6

[nbruin]

comment:7

Not really ready for review but is ready for input. And is mainly ready for someone else to rewrite documentation properly. People CC'd on this ticket look like good candidates to do so.

---

New commits:

b41248b	trac 17447: Deprecation of function('f',x) in favour of function('f')(x)
fa3ebd6	trac 17447: Doctest changes to reflect deprecation of function('f',x)

---

note that the doctest that apparently got removed in b41248b in reality gets moved to var.pyx because it tests the toplevel function, not the library function

[94a1bc6f-6005-414c-9598-64422d48cc95]

comment:8

Replying to @nbruin:

Not really ready for review but is ready for input. And is mainly ready for someone else to rewrite documentation properly. People CC'd on this ticket look like good candidates to do so.

---

New commits:
b41248b	trac 17447: Deprecation of function('f',x) in favour of function('f')(x)
fa3ebd6	trac 17447: Doctest changes to reflect deprecation of function('f',x)

---

note that the doctest that apparently got removed in b41248b in reality gets moved to var.pyx because it tests the toplevel function, not the library function

This makes it a lot clearer to me now. Just a minor thing: In one of the examples, I would replace

sage: cr = function('cr')
sage: f = cr(a)

by

sage: function('cr')
sage: f = cr(a)

which in my understanding does the same but is shorter and makes clear that function('cr') does already create a symbolic function called cr. Otherwise, one may be surprised to create two functions by typing e.g. cr1 = function('cr').

[nbruin]

comment:9

Replying to @sagetrac-schymans:

This makes it a lot clearer to me now. Just a minor thing: In one of the examples, I would replace

sage: cr = function('cr')
sage: f = cr(a)

At least one occurrence of that is in the doctest of sage.symbolic.function_factory.function, which (now) goes out of its way to import that function, which does not have the side-effect of mutating the global scope, as documented. So the assignment is actually necessary.

Another independent point: the top-level function does refer to sage.symbolic.function_factory.function, but perhaps should do so more explicitly, if possible with a hyperlink. The examples on the latter are a little more elaborate, so someone who wants to read up on function (and would probably find the top-level documentation first via sage: function?). I don't know how to make hyperlinks to other documentation in sage's docstrings.

[6bdad4c1-1e26-4f2f-a442-a01a2292c181]

comment:10

There are broken doctests in the 'doc' folder

sage -t --long prep/Quickstarts/Differential-Equations.rst  # 1 doctest failed
sage -t --long tutorial/tour_algebra.rst  # 1 doctest failed
sage -t --long constructions/calculus.rst  # 1 doctest failed

Nathann

[nbruin]

comment:11

Replying to @nathanncohen:

There are broken doctests in the 'doc' folder

Yes, and that is not the only place. First we need to see if there is any good reason why function('f',x) is preferable over function('f')(x), of if we can get away with deprecating this confusing construction. And then someone should take a look at the doc to see if it needs further improvement. You're on a doc revamping binge, so this might just be the task for you!

[6bdad4c1-1e26-4f2f-a442-a01a2292c181]

comment:12

You're on a doc revamping binge, so this might just be the task for you!

I am afraid that you will have to find somebody else. I never used Sage's symbolics.

Nathann

[rwst]

Changed branch from u/nbruin/clarify_and_complete_documentation_of_function__ to public/17447

[rwst]

comment:15

That fixes the three above marked doctests.

---

New commits:

e3ef919	Merge branch 'develop' into t/17447/clarify_and_complete_documentation_of_function__
e6a3534	17447: reviewer's patch: fix rst doctests

[rwst]

Changed commit from fa3ebd6 to e6a3534

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

4585e74

17447: fix rest of doctests

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from e6a3534 to 4585e74

[rwst]

Reviewer: Ralf Stephan

[rwst]

comment:17

These are the remaining doctest fixes coming from make ptestlong. I have reviewed and found fine the commits up to mine, so following reviewers can skip that.

And then someone should take a look at the doc to see if it needs further improvement.

[kcrisman]

comment:18

I suspect that Zimmerman's calculus book uses this stuff quite extensively, so deprecating/changing the current behaviour will likely lead to pushback from him, and probably rightly so.

Cc:ing him for that reason. Though we have already had a few other discussions on Trac about the need to update our tests while not maintaining exact compatibility. Since they don't (yet) raise errors, but apparently only the deprecation warning, should we maybe update the tests (in that part only) to have the deprecation warning returned? Note that I don't believe the deprecation warning is even doctested, which is a no-no :)

Also, does the current branch actually "clarify and complete documentation of function"? It looks like it mostly fixes doctests.

Another change not doctested is

     if is_SymbolicVariable(dvar):
-        raise ValueError("You have to declare dependent variable as a function, eg. y=function('y',x)")
+        raise ValueError("You have to declare dependent variable as a function evaluated at the independent variable, eg. y=function('y')(x)")

which I think happens twice.

I'm still not sure I even understand some of the subtle differences. Are there occasions where the old behavior was "right" in the sense that one wanted that returned, and have we shown how to get that object? What about the ask.sagemath question?

All that to say that nonetheless it will be great to have a unified interface on this, if that is really the right thing to do, which from the comments it apparently is.

[nbruin]

comment:19

Replying to @kcrisman:

Also, does the current branch actually "clarify and complete documentation of function"? It looks like it mostly fixes doctests.

I think there are two things that are/were confusing:

• function('f',x) returns something that is not, according to our definition, a function (e.g., it cannot be evaluated ("called") in a non-ambiguous way and hence triggers a deprecation warning)).
• The top-level function makes a binding in the global namespace and returns a value. The two were different.

With the proposed patch, function('f',x) is deprecated, so number 1 gets eliminated and number 2 is less confusing because, while it still injects something in the namespace, the value it returns is always the same as what it injects.

When I read the documentation of sage.symbolic.function_factory.function, I actually thought it described what happens fairly correctly, and it has good examples. Hence I did not feel the need to add to it. The top-level function documentation is rather sparse, so that could use expansion. However, we do not want to repeat ourselves, so perhaps it should just contain a clear pointer (hyperlink?) to the documentation of sage.symbolic.function_factory.function. I don't know how to do that, so please go ahead and improve that bit. You are excellently qualified to produce extensive and understandable documentation for the target audience of these functions.

Another change not doctested is

     if is_SymbolicVariable(dvar):
-        raise ValueError("You have to declare dependent variable as a function, eg. y=function('y',x)")
+        raise ValueError("You have to declare dependent variable as a function evaluated at the independent variable, eg. y=function('y')(x)")

which I think happens twice.

These messages weren't doctested before either. If you feel the projects benefits from such administrations, go ahead.

I'm still not sure I even understand some of the subtle differences. Are there occasions where the old behavior was "right" in the sense that one wanted that returned, and have we shown how to get that object?

In my opinion: no. In all cases, function('f')(x) is much clearer and not much longer than function('f',x). There are many tickets and questions about people getting tripped up by the fact that a function called function doesn't return a function.

What about the ask.sagemath question?

In my opinion, that's another thing to document. However, the confusion about function not returning a function definitely contributed to the confusion about how to use substitute_function.

[kcrisman]

comment:20

The top-level function documentation is rather sparse, so that could use expansion.

Yes, this is what I meant. Hyperlink is tricky - not per se, but in command line and even sagenb it's not as useful. So I figure at least a little more in the top-level and then the hyperlink (assuming that file is in fact already in the documentation!).

You are excellently qualified to produce extensive and understandable documentation for the target audience of these functions.

That is very kind, I will see; this one is lower on my priority list but definitely important.

If you feel the projects benefits from such administrations, go ahead.

Well, I've found it does. I guess I will ask you to review it if I do that :-)

In my opinion, that's another thing to document. However, the confusion about function not returning a function definitely contributed to the confusion about how to use substitute_function.

Yes.

[zimmermann6]

comment:21

Replying to @kcrisman:

I suspect that Zimmerman's calculus book uses this stuff quite extensively, so deprecating/changing the current behaviour will likely lead to pushback from him, and probably rightly so.

Cc:ing him for that reason.

thanks, however I gave up trying to maintain compatibility with the book, since there are already many examples from the book that we put as doctests in Sage, and which were changed afterwards...

[kcrisman]

comment:22

Here is another example of something confusing.

http://ask.sagemath.org/question/26114/why-is-basic-arithmetic-disallowed-on-symbolic-functions/

[nbruin]

comment:23

Replying to @kcrisman:

Here is another example of something confusing.

http://ask.sagemath.org/question/26114/why-is-basic-arithmetic-disallowed-on-symbolic-functions/

I think that's another issue. The problem there is that global namespace var and function return a value as well as insert something into the namespace. That's counter to python custom (routines that mutate state normally return None. Compare L.sort() and sorted(L). There are exceptions: L.pop()).

Initially it seems pedantic to enforce such rules in a computer algebra system as well, but the many problems it causes suggests it's not. Can we have declare_var('x,y') and declare_function('f') for the mutating stuff and just have var('x') and function('f') for the normal stuff? That ship has probably sailed already.

[94a1bc6f-6005-414c-9598-64422d48cc95]

comment:24

Replying to @nbruin:

Can we have declare_var('x,y') and declare_function('f') for the mutating stuff and just have var('x') and function('f') for the normal stuff? That ship has probably sailed already.

That would be awesome! Much more consistent AND flexible. +1 from me.

[rwst]

comment:25

Replying to @nbruin:

I think that's another issue. The problem there is that global namespace var and function return a value as well as insert something into the namespace. That's counter to python custom (routines that mutate state normally return None. Compare L.sort() and sorted(L). There are exceptions: L.pop()).

Initially it seems pedantic to enforce such rules in a computer algebra system as well, but the many problems it causes suggests it's not. Can we have declare_var('x,y') and declare_function('f') for the mutating stuff and just have var('x') and function('f') for the normal stuff? That ship has probably sailed already.

See #17958 for declare_var and no, this is still a good idea.

[mezzarobba]

comment:26

Replying to @nbruin:

I suspect that Zimmerman's calculus book uses this stuff quite extensively, so deprecating/changing the current behaviour will likely lead to pushback from him, and probably rightly so.

I'm not sure about function() and have little time to check now, but regarding var(), I think we tried to always use x = SR.var('x').

And IMO, the version of var that injects a variable in the global namespace should simply be deprecated, or possibly moved to sage.ext.inteactive_constructors_c if someone really care about it.

[rwst]

comment:27

So, comment:17 gave part of a review. Can someone please complete it?

[rwst]

comment:28

sage -t --long src/sage/symbolic/expression.pyx  # 1 doctest failed
sage -t --long src/sage/tests/french_book/calculus_doctest.py  # 1 doctest failed
sage -t --long src/doc/pt/tutorial/tour_algebra.rst  # 1 doctest failed
sage -t --long src/sage/combinat/integer_vector.py  # 1 doctest failed

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

7956412	Merge branch 'public/17447' of git://trac.sagemath.org/sage into public/17447
bdc1543	trac 17447: further doctest adjustments

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 4585e74 to bdc1543

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

7c029f3

trac 17447: further doctest adjustments

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from bdc1543 to 7c029f3

[nbruin]

comment:32

OK, comments #17 and #27 provide a positive review of the first bit. I give a largely positive review to Ralf's doctest adjustments. There was one overcorrection in french_book/calculus_doctest.py that I corrected. The other changes were just parallel corrections to different translations of one document that have been introduced since the last time this ticket was updated.

patchbot is happy and I think all code is positively reviewed now. Ready for merge!

[nbruin]

Changed author from schymans to Nils Bruin, Ralf Stephan

[nbruin]

Changed reviewer from Ralf Stephan to Ralf Stephan, Nils Bruin

[vbraun]

Changed branch from public/17447 to 7c029f3