locate and code for functions involving options, caching, or @@ are useless

[mahrud]

Say I want to see where net Variety is defined, so I use locate:

i5 : locate(net, Variety)

o5 = (../../Macaulay2/m2/classes.m2, 90, 47, 90, 54, 90, 50)

But that doesn't make sense, so instead I look up its code:

i8 : code(net, Variety)

o8 = -- code for method: net(Variety)
     ../../Macaulay2/m2/classes.m2:90:48-90:55: --source code:
     Function @@ Function := Function => (f,g) -> x -> f g x
     | symbol   class                   value      location of symbol
     | ------   -----                   -----      ------------------                       
     | f      : MethodFunctionSingle -- net        ../../Macaulay2/m2/classes.m2:90:38-90:39
     | g      : MethodFunctionSingle -- expression ../../Macaulay2/m2/classes.m2:90:40-90:41
     | -- function f:
     | function net: source code not available
     | -- function g:
     | function expression: source code not available

But this is useless: nowhere in this output does it say that net Variety is defined on line 24 of varieties.m2, or even what net and expression do or even where they are defined -- it just says the are also in classes.m2! Even the most useful part of this output, which says that net Variety is the composition of net and expression, is written in too much generality and would be much easier to understand if it just said something like x -> net expression x.

To a smaller extent, the same is true for functions involving options:

i17 : locate(codim, ProjectiveVariety)

o17 = (../../Macaulay2/m2/option.m2, 15, 19, 17, 33, 17, 7)

o17 : Sequence

i18 : code(codim, ProjectiveVariety)

o18 = -- code for method: codim(ProjectiveVariety)
      ../../Macaulay2/m2/option.m2:15:20-19:34: --source code:
        (opts,f) -> args -> (
             -- Common code for functions created with >> to process options and arguments.
             uncurry(f, override (opts,args))
             )
        )
      | symbol   class              value                         location of symbol
      | ------   -----              -----                         ------------------                     
      | f      : FunctionClosure -- ...                           ../../Macaulay2/m2/option.m2:15:9-15:10
      | opts   : OptionTable     -- OptionTable{Generic => false} ../../Macaulay2/m2/option.m2:15:4-15:8 
      | -- function f:
      | ../../Macaulay2/m2/varieties.m2:299:66-299:87: --source code:
      | codim ProjectiveVariety := options(codim,PolynomialRing) >> opts -> X -> codim(ring X,opts)
      | -- option table opts:
      | OptionTable{Generic => false}

and those involving caching:

i36 : locate(poincare,MonomialIdeal)

o36 = (../../Macaulay2/m2/methods.m2, 654, 51, 663, 25, 655, 56)

o36 : Sequence

i37 : code(poincare, MonomialIdeal)

o37 = -- code for method: poincare(MonomialIdeal)
      ../../Macaulay2/m2/methods.m2:654:52-663:26: --source code:
      cacheValue = key -> f -> new CacheFunction from (x -> (
                c := try x.cache else x.cache = new CacheTable;
                if c#?key then (
                     val := c#key;
                     if class val === CacheFunction then (
                          remove(c,key);
                          c#key = val x)
                     else val
                     )
                else c#key = f x))
      | symbol   class              value    location of symbol
      | ------   -----              -----    ------------------                         
      | f      : FunctionClosure -- ...      ../../Macaulay2/m2/methods.m2:654:21-654:22
      | key    : Symbol          -- poincare ../../Macaulay2/m2/methods.m2:654:14-654:17
      | -- function f:
      | ../../Macaulay2/m2/monideal.m2:98:59-98:123: --source code:
      | poincare MonomialIdeal := (cacheValue symbol poincare) (M -> new degreesRing M from rawHilbert rawMonomialIdealToMatrix M.RawMonomialIdeal)

Together, this makes the output of code methods annoyingly long.

[mahrud]

Oh my goodness this is the worst:

i1 : code(options, PolynomialRing)

o1 = -- code for method: options(PolynomialRing)
     ../linuxbrew/.linuxbrew/share/Macaulay2/Core/classes.m2:90:48-90:55: --source code:
     Function @@ Function := Function => (f,g) -> x -> f g x
     | symbol   class                   value   location of symbol
     | ------   -----                   -----   ------------------                                                 
     | f      : MethodFunctionSingle -- options ../linuxbrew/.linuxbrew/share/Macaulay2/Core/classes.m2:90:38-90:39
     | g      : MethodFunctionSingle -- monoid  ../linuxbrew/.linuxbrew/share/Macaulay2/Core/classes.m2:90:40-90:41
     | -- function f:
     | function options: source code not available
     | -- function g:
     | ../linuxbrew/.linuxbrew/share/Macaulay2/Core/option.m2:15:20-19:34: --source code:
     |   (opts,f) -> args -> (
     |        -- Common code for functions created with >> to process options and arguments.
     |        uncurry(f, override (opts,args))
     |        )
     |   )
     | | symbol   class              value                                                   location of symbol
     | | ------   -----              -----                                                   ------------------                                               
     | | f      : FunctionClosure -- ...                                                     ../linuxbrew/.linuxbrew/share/Macaulay2/Core/option.m2:15:9-15:10
     | | opts   : OptionTable     -- OptionTable{Constants => false                        } ../linuxbrew/.linuxbrew/share/Macaulay2/Core/option.m2:15:4-15:8 
     | |                                         DegreeLift => null                          
     | |                                         DegreeMap => null                           
     | |                             ....................................................... 
     | | -- function f:
     | | ../linuxbrew/.linuxbrew/share/Macaulay2/Core/methods.m2:144:33-146:100: --source code:
     | |     methodFunction := opts >> o -> arg -> (
     | |         -- Common code for every method with options and a single argument
     | |         singleCaller(methodFunction, (methodFunction, dispatchBy arg), arg, outputs, dispatcher(o, arg)));
     | | | symbol           class                   value                                                   location of symbol
     | | | ------           -----                   -----                                                   ------------------                                                   
     | | | opts           : OptionTable          -- OptionTable{Constants => false                        } ../linuxbrew/.linuxbrew/share/Macaulay2/Core/methods.m2:136:23-136:27
     | | |                                                      DegreeLift => null                          
     | | |                                                      DegreeMap => null                           
     | | |                                          ....................................................... 
     | | | dispatchBy     : CompiledFunction     -- class                                                   ../linuxbrew/.linuxbrew/share/Macaulay2/Core/methods.m2:140:5-140:15 
     | | | dispatcher     : FunctionClosure      -- ...                                                     ../linuxbrew/.linuxbrew/share/Macaulay2/Core/methods.m2:141:5-141:15 
     | | | outputs        : Boolean              -- false                                                   ../linuxbrew/.linuxbrew/share/Macaulay2/Core/methods.m2:136:29-136:36
     | | | methodFunction : MethodFunctionSingle -- monoid                                                  ../linuxbrew/.linuxbrew/share/Macaulay2/Core/methods.m2:144:5-144:19 
     | | -- option table opts:
     | | OptionTable{Constants => false                        }
     | |             DegreeLift => null
     | |             DegreeMap => null
     | |             DegreeRank => null
     | |             Degrees => null
     | |             Global => true
     | |             Heft => null
     | |             Inverses => false
     | |             Join => null
     | |             Local => false
     | |             MonomialOrder => {GRevLex, Position => Up}
     | |             MonomialSize => 32
     | |             SkewCommutative => {}
     | |             VariableBaseName => p
     | |             Variables => null
     | |             Weights => {}
     | |             WeylAlgebra => {}

I have no idea what prompted all of this! Here is the actual code in ofcm.m2

options PolynomialRing := options @@ monoid

Nowhere in all of that text is ofcm.m2 mentioned.

[DanGrayson]

The point is that in the line options PolynomialRing := options @@ monoid, the code to the right of the := is executed, not saved for later execution. If you'd like it to be clearer, just change it to options PolynomialRing := R -> options monoid R.

[mahrud]

That has nothing to do with locate not showing where (options, Polynomial) is set to the value of whatever the right hand side evaluates to, or code showing all that extra stuff.

[DanGrayson]

No, locate operates on code stored in function bodies -- that's where location information is stored.

[mahrud]

Okay ... sounds like you agree with what I'm claiming in this issue, then? That "locate and code for functions involving options, caching, or @@ are useless"?

[mikestillman]

Is there any easy way we can get these to work (and not be 'useless')?

[mikestillman]

Probably the file/line location of the actual method setting should be given instead of where the function is defined?

[mahrud]

If the execution stack was somehow accessible (which is another feature request on its own), then I think the easiest way would be for Function @@ Function to store location of the one before the last value in the execution stack, which would be the line that called Function @@ Function. If the location is stored in the function frame then it could be accessed later.

[DanGrayson]

options PolynomialRing := options @@ monoid

Yes, I agree that you cannot find the line options PolynomialRing := options @@ monoid by using locate. However, you could change that line to options PolynomialRing := R -> options monoid R.

I find it easy to use emacs' tags search function to find the line options PolynomialRing := options @@ monoid in the code.

[mahrud]

So you're suggesting just getting rid of @@ all over the place? That seems like erasing the problem statement to me.

Hooks are also kind of opaque, you can't tell what's in the code from code(length, Module), for instance, but if you actually want to make it useful (instead of just saying "well don't use hooks" or "use emacs") then there's a solution:

i1 : code(length, Module)

o1 = -- code for method: length(Module)
     ../linuxbrew/.linuxbrew/share/Macaulay2/Core/hilbert.m2:148:26-151:89: --source code:
     length Module := ZZ => M -> (
         computation := (cacheValue symbol length) (M -> runHooks((length, Module), M));
         if (n := computation M) =!= null then return n;
         error("no applicable strategy for computing length of modules over ", toString ring M))

i2 : code hooks(length, Module)

o2 = -- code for method: length(Module)
     ../linuxbrew/.linuxbrew/share/Macaulay2/Core/hilbert.m2:153:50-155:44: --source code:
     addHook((length, Module), Strategy => Default, M -> (
         if not isHomogeneous M then notImplemented();
         if dim M > 0 then infinity else degree M))

[DanGrayson]

So you're suggesting just getting rid of @@ all over the place? That seems like erasing the problem statement to me.

This is just like the code x = 2 + 2. If later on, you have that number 4, there is no way to use it to find out the location of the code 2 + 2 that resulted in it.

[mahrud]

This is just like the code x = 2 + 2. If later on, you have that number 4, there is no way to use it to find out the location of the code 2 + 2 that resulted in it.

1. How is this relevant? I'm talking about method functions, not symbols.
2. You can still find where the symbol x was created with locate x.

I'm not surprised that you disagree with my improvement suggestions, but you don't need to keep insisting.

[DanGrayson]

This is just like the code x = 2 + 2. If later on, you have that number 4, there is no way to use it to find out the location of the code 2 + 2 that resulted in it.

1. How is this relevant? I'm talking about method functions, not symbols.

It's relevant because both x = 2 + 2 and options PolynomialRing := options @@ monoid are assignment statements. In both of them, the right hand side is evaluated, and the value is stored in the location indicated by the left hand side. The location of the code that computes the value is not stored in the value.

There are other ways to produce functions -- not just f @@ g. How could you handle all of them uniformly?

[mahrud]

Sure, there are many ways to produce functions, but the three classes of functions I'm highlighting in this issue are extremely common, so even ad hoc solutions (e.g. changing Function @@ Function as I suggested above) would go a long way.

But if you must have a general solution: the left hand side of options PolynomialRing := X also evaluates something, and that code can query the current location and store it in the function produced by the right hand side, however it is produced. This wouldn't be too hard with some internal tweaking to make the register storing the location of a function mutable.

[mahrud]

This is just like the code x = 2 + 2. If later on, you have that number 4, there is no way to use it to find out the location of the code 2 + 2 that resulted in it.

I'm reading the bold part again, and it seems even less relevant. In all the cases I highlighted locate and code don't start with a function, they start with a method sequence like (options, Polynomial).

[mikestillman]

It seems to me that if one has a method definition, such as

f = method()
f Matrix := g

where g is a function, then code(f, Matrix) (or locate?) could give one of two locations: the location of the line f Matrix := g, or the location of the definition of g. It seems like the former might be more useful information?

[mahrud]

I think both should be present in the output of code, at least.

[DanGrayson]

But if you must have a general solution: the left hand side of options PolynomialRing := X also evaluates something, and that code can query the current location and store it in the function produced by the right hand side, however it is produced. This wouldn't be too hard with some internal tweaking to make the register storing the location of a function mutable.

That might work. But don't store it in the function, as one still wants to be able to find the location of the function itself. Also, the same function might be installed as multiple method functions. So the location information should be stored alongside the function in one of the types on the left hand side.

[mahrud]

Here's another case where code not only has a lot of random output but is literally useless: I noticed there's a method Constant ^ Ring, which is not documented and somehow has options (but how do you even pass options to this?!) so I wanted to look at the code, but this is what I got:

i1 : code(symbol^, Constant, Ring)

o1 = -- code for method: Constant ^ Ring
     /home/linuxbrew/.linuxbrew/share/Macaulay2/Core/option.m2:15:19-17:33: --source code:
       (opts,f) -> args -> (
            -- Common code for functions created with >> to process options and arguments.
            uncurry(f, override (opts,args))
            )
       )
     | symbol  class            value                                                    location of symbol
     | ------  -----            -----                                                    ------------------                                                 
     | f       FunctionClosure  FunctionClosure[/home/linuxbrew/.linuxbrew/share/Macau.  /home/linuxbrew/.linuxbrew/share/Macaulay2/Core/option.m2:15:8-15:9
     | opts    OptionTable      OptionTable{Verify => true}                              /home/linuxbrew/.linuxbrew/share/Macaulay2/Core/option.m2:15:3-15:7
     | -- function f:
     | /home/linuxbrew/.linuxbrew/share/Macaulay2/Core/methods.m2:154:69-154:101: --source code:
     |      methodFunction := new MethodFunctionWithOptions from (opts >> o -> arg -> innerMethodFunction(o,arg));
     | | symbol               class                      value                                         location of symbol
     | | ------               -----                      -----                                         ------------------                                                      
     | | opts                 OptionTable                OptionTable{Verify => true}                   /home/linuxbrew/.linuxbrew/share/Macaulay2/Core/methods.m2:151:37-151:41
     | | innerMethodFunction  CompiledFunctionClosure    CompiledFunctionClosure[]                     /home/linuxbrew/.linuxbrew/share/Macaulay2/Core/methods.m2:153:11-153:30
     | | outputs              List                       {false, true, true}                           /home/linuxbrew/.linuxbrew/share/Macaulay2/Core/methods.m2:151:42-151:49
     | | methopts             OptionTable                OptionTable{Binary => false                }  /home/linuxbrew/.linuxbrew/share/Macaulay2/Core/methods.m2:151:28-151:36
     | |                                                             Dispatch => {Thing, Type, Type}   
     | |                                                             Options => {Verify => true}       
     | |                                                             TypicalValue => Thing             
     | | methodFunction       MethodFunctionWithOptions  lift                                          /home/linuxbrew/.linuxbrew/share/Macaulay2/Core/methods.m2:154:5-154:19 
     | -- option table opts:
     | OptionTable{Verify => true}

This gives me zero information, other than a mysterious lift near the end. Even the place where f is defined is obscured:

     | f       FunctionClosure  FunctionClosure[/home/linuxbrew/.linuxbrew/share/Macau.  /home/linuxbrew/.linuxbrew/share/Macaulay2/Core/option.m2:15:8-15:9

[mahrud]

The most egregious case I've ran into so far:

ii8 : code(symbol //, RingElement, ZZ)

oo8 = -- code for method: RingElement // ZZ
      /home/mahrud/Projects/M2/feature/M2/Macaulay2/m2/classes.m2:106:47-106:54: --source code:
      Function @@ Function := Function => (f,g) -> x -> f g x
      | symbol  class            value              location of symbol
      | ------  -----            -----              ------------------                                                       
      | f       FunctionClosure  quot0              /home/mahrud/Projects/M2/feature/M2/Macaulay2/m2/classes.m2:106:37-106:38
      | g       FunctionClosure  promoterightexact  /home/mahrud/Projects/M2/feature/M2/Macaulay2/m2/classes.m2:106:39-106:40
      | -- function f:
      | /home/mahrud/Projects/M2/feature/M2/Macaulay2/m2/enginering.m2:514:14-514:22: --source code:
      | quot0 = (f,g) -> f // g
      | -- function g:
      | /home/mahrud/Projects/M2/feature/M2/Macaulay2/m2/classes.m2:106:47-106:54: --source code:
      | Function @@ Function := Function => (f,g) -> x -> f g x
      | | symbol  class            value                                                    location of symbol
      | | ------  -----            -----                                                    ------------------                                                       
      | | f       FunctionClosure  FunctionClosure[/home/mahrud/Projects/M2/feature/M2/Ma.  /home/mahrud/Projects/M2/feature/M2/Macaulay2/m2/classes.m2:106:37-106:38
      | | g       FunctionClosure  swap                                                     /home/mahrud/Projects/M2/feature/M2/Macaulay2/m2/classes.m2:106:39-106:40
      | | -- function f:
      | | /home/mahrud/Projects/M2/feature/M2/Macaulay2/m2/classes.m2:106:47-106:54: --source code:
      | | Function @@ Function := Function => (f,g) -> x -> f g x
      | | | symbol  class            value             location of symbol
      | | | ------  -----            -----             ------------------                                                       
      | | | f       FunctionClosure  swap              /home/mahrud/Projects/M2/feature/M2/Macaulay2/m2/classes.m2:106:37-106:38
      | | | g       FunctionClosure  promoteleftexact  /home/mahrud/Projects/M2/feature/M2/Macaulay2/m2/classes.m2:106:39-106:40
      | | | -- function f:
      | | | /home/mahrud/Projects/M2/feature/M2/Macaulay2/m2/enginering.m2:465:13-465:19: --source code:
      | | | swap = (x,y) -> (y,x)
      | | | -- function g:
      | | | /home/mahrud/Projects/M2/feature/M2/Macaulay2/m2/enginering.m2:462:25-464:8: --source code:
      | | | promoteleftexact = (f,g) -> (
      | | |      f = try promote(f,class g) else oops();
      | | |      (f,g))
      | | -- function g:
      | | /home/mahrud/Projects/M2/feature/M2/Macaulay2/m2/enginering.m2:465:13-465:19: --source code:
      | | swap = (x,y) -> (y,x)