Context switching doesn't work in some cases

[vsht]

Here is a sample code (used e.g. in FeynCalc) that allows to define a variable in some context inside a package in such a way, that it can directly accessed from the subpackages

BeginPackage["FeynCalc`"];

AppendTo[$ContextPath, "FeynCalc`Package`"];

Begin["`Package`"];
(*Symbols to be shared between subpackages*)
sharedSymbol;
End[];

foo::usage = "";
bar::usage = "";

(*-----------------------------------------*)

Begin["`MySubpackageA`Private`"];
intVarA::usage = "";
foo[x_] := (
   Print["I can access sharedSymbol directly, since it is in ", 
    Context[sharedSymbol], " and not in ",
    Context[intVarA]];
   sharedSymbol = x;
   x
   );
End[];

(*-----------------------------------------*)

Begin["`MySubpackageB`Private`"];
intVarB::usage = "";
bar[] := (
   Print["I can access sharedSymbol directly, since it is in ", 
    Context[sharedSymbol], " and not in ",
    Context[intVarB]];
   sharedSymbol
   );
End[];

EndPackage[];

The idea is that the code of MySubpackageA and MySubpackageB is loaded from external files but this loading still takes place inside

BeginPackage["FeynCalc`"];
...
EndPackage[];

By adding "FeynCalc`Package" to the context path I can introduce sharedSymbol in

Begin["`Package`"];
(*Symbols to be shared between subpackages*)
sharedSymbol;
End[];

in such a way, that MySubpackageA and MySubpackageB can access it directly instead of writing out the full context path which is

FeynCalc`Package`sharedSymbol

Here is an example in Mma

sharedContext.zip

Get["sharedContext.m"]
foo[42]
bar[]

In[1]:= Get["sharedContext.m"]                                                                                                                                                                                                                                                              

In[2]:= foo[42]                                                                                                                                                                                                                                                                             
I can access sharedSymbol directly, since it is in FeynCalc`Package` and not in FeynCalc`MySubpackageA`Private`

Out[2]= 42

In[3]:= bar[]                                                                                                                                                                                                                                                                               
I can access sharedSymbol directly, since it is in FeynCalc`Package` and not in FeynCalc`MySubpackageB`Private`

Out[3]= 42

For some reason this doesn't work in Mathics

In[1]:= Get["sharedContext.m"]
In[2]:= foo[42]
I can access sharedSymbol directly, since it is in FeynCalc`MySubpackageA`Private` and not in FeynCalc`MySubpackageA`Private`
Out[2]= 42

In[3]:= bar[]
I can access sharedSymbol directly, since it is in FeynCalc`MySubpackageB`Private` and not in FeynCalc`MySubpackageB`Private`
Out[3]= FeynCalc`MySubpackageB`Private`sharedSymbol

This is highly relevant for #1174

[mmatera]

After some tests, I realized that the problem here is that you do not assign anything to sharedSymbol in the first call. The code does what is expected if you write for example

Begin["`Package`"];
(*Symbols to be shared between subpackages*)
sharedSymbol=Null;
End[];

I can look for the possibility of creating the symbol just by calling it, but I think that assign Null to this kind of symbols fix the problem and also is more clear. @vsht, @rocky, what do you think?

[rocky]

@mmatera I don't know enough to have an informed opinion. The question I'd ask @vsht is whether this example, which has been stripped down from something else, really represents what is needed. Or is it instead stripped down more than the basic need? But also whether the use is assuming unspecified behavior getting implemented in a particular way, since an implemenation can have a deterministic behavior, whether or not it has been sanctioned and specified.

Here is an analogous situation that you often find in Python. The keys in Python2 dictionaries and sets are not guarenteed to follow a particular order. But, in practice, they do follow a particular order because the implementation uses the same hashing algorithm inside the Python interpreter and no effort has been made (or was desired) to randomize the hash between successive calls. In Python3 the hashing algorithm changed and so the order of the same set of values is different between Python2 and Python3. It is not uncommon for programs to assume that the arbitrary order will stay as the one that was given when when the Python programmer wrote the code. As a result, programs wriiten for Python2 that make use of Python2's set order (determined by the hashing function that is internally used) can fail on Python3 and vice versa. But the Python implementers are happy to inform you that your program is at fault, not Python2 or Python3.

So similarly is there a possiblity that the implementation of WMA might change its default behavior that might break the expected behavior or the test case or more generally of the code that is really needed because it is making an assumption about ill-defined behavior?

As for what the answer is and what should be done to Mathics, I leave in you both since, again, you are in a much better position to know, understand, and advise.

[vsht]

I don't really see why this example should be considered unspecified behavior.

Begin and End are two generic commands for switching the context and so one can use them in multiple ways inside and outside of a package. There is nothing illegal about that. The usage case I provided is actually used by Mma iself, e.g. in the JLink subpackage (have a look at SystemFiles/Links/JLink/Kernel).

I don't know how Mathics currently treats such commands, so it's not clear to me why my example doesn't work. If you can explain the current implementation and the underlying assumptions, one could surely compare that to the default behavior of Mma.

[mmatera]

This does not have anything to do with Begin and End: the problem is the line sharedSymbol; In the current implementation (in master), this does not add sharedSymbol in the current context to the dictionary of user definitions. To get what you expect, you execute instead sharedSymbol=Null;

In #1305, I propose a change of this behaviour, in a way that
sharedSymbol; adds an empty definition in the current context, so you get the expected behaviour in WMA

[vsht]

Thanks for the explanation, now I understand the problem better. Well, Mma always adds any symbol you evaluate to the Global context, even if you don't assign it any value. Consider e.g. the following

xxx;
ValueQ[xxx]
Names["Global`*"]

MMa:

In[1]:= xxx;                                                                                                                                 

In[2]:= ValueQ[xxx]                                                                                                                          

Out[2]= False

In[3]:= Names["Global`*"]                                                                                                                    

Out[3]= {xxx}

Mathics:

In[1]:= xxx;
In[2]:= ValueQ[xxx]
Out[2]= False

In[3]:= Names["Global`*"]
Out[3]= {}

This is way the Global context quickly gets polluted by useless junk, but unfortunately this is how Mathematica works internally. Admittedly, for aesthetic reasons I like the bevavior Mathics better, but to be compatible with Mma codes it probably should better behave the way Mma does.

[mmatera]

This is the reason for my question: on the one hand, it is really nice that you put a slice of salami on WMA and you get the salami evaluated but, on the other hand, it does not helps to compel the users to write clear code. If this kind of "incompatibility" forces the users to write clearer code, which will be useable on both implementations, maybe I would keep the things just as they are now. On the other hand, it wasn't too hard to reproduce the WMA behaviour. This is why I am asking for opinions.

[vsht]

Well, as shown in my original example, one can exploit this behavior of Mma to achieve particular behavior of symbols in the packages (e.g. sorting them into desired contexts).

WL is not a standardized language, so I would say that the behavior of Mma that remains canonical over multiple versions can be very much considered to be the default.

If this kind of "incompatibility" forces the users to write clearer code, which will be useable on both implementations, maybe I would keep the things just as they are now.

I see three possible issues with this approach:

1. If you insist on keeping things not 100% compatible in principle, potential users may be scared away
2. Codes that work on Mathics without any issues may run into shadowing problems when evaluated on Mma. This would be some kind of reverse compatibility problem.
3. WL per se allows you to write codes that are very difficult to read. Think of all those //, @, @@ and & symbols that essentially allows you to convert 20 different commands into one fat one-liner. The language doesn't enforce a clear code formatting like e.g. Python so that there is hardly a way to avoid mess if one wants to be messy.

On the other hand, it wasn't too hard to reproduce the WMA behaviour. This is why I am asking for opinions.

Perhaps implement it as an option inside SystemOptions?

Anyway, I'm not a member of this project, so I can only voice my opinion, but at the end the developers decide what gets changed and what not.

[rocky]

@vsht Do I have this correct?

1. The behavior for the idiom used is not specified
2. Rewriting FeynCalc is a way that avoids the unspecified behavior is doable but make things a lot less convenient or downright ugly

If that's the case, personally I have no problem being compatible with unspecified behavior. But really the final decision for @mmatera since he's been doing the work.

[mmatera]

Please, do not misunderstand me: the fix is ready, and it does not imply very drastic changes in the code. My unique concern is that our current (active) development team are @rocky and me, and (as Houllebecq says) a democracy works better with an prime odd number of electors.

[mmatera]

By the way, @vsht, I start to include some of your tests in the test engine of the project, so technically, to appear as a part of the team is up to you :)

[vsht]

Please, do not misunderstand me: the fix is ready, and it does not imply very drastic changes in the code. My unique concern is that our current (active) development team are @rocky and me, and (as Houllebecq says) a democracy works better with an prime odd number of electors.

No problem. Actually, now that I looked it up, I think that this behavior is even documented. Have a look at https://reference.wolfram.com/language/tutorial/ModularityAndTheNamingOfThings.html#25032

It says

The Wolfram Language creates a new symbol when you first enter a particular name.

This is also demonstrated in one of the examples in that section, e.g.

$NewSymbol=Print["Name:", #1, " Context: ", #2]& 
v+w

In this sense, writing xyz; is a well-defined way to tell Mma to create xyz in the current context without assigning it any value.

By the way, @vsht, I start to include some of your tests in the test engine of the project, so technically, to appear as a part of the team is up to you :)

The problem is time. I'm always short of it, since there are always too many things to do (especially academia-wise) and everything is somehow super urgent and to be finished ASAP ...

[mmatera]

The problem is time. I'm always short of it, since there are always too many things to do (especially academia-wise) and everything is somehow super urgent and to be finished ASAP ...

Yes, I understand. Normally I have the same problem, but now, due to the isolation, many of these things have to wait anyway, and during the day, working on tasks that demand much attention is very hard being at home with my daughter, so I spend some extra time on this, that does not require too much concentration :)