Static options/variables and scoping

[ndw]

I don't really understand this paragraph:

Neither options nor variables in the prologue may shadow each other. It is a static error (err:XS0091) if a p:option or p:variable declared before the subpipeline begins shadows another option or variable declared within the same p:declare-step. (Within the subpipeline, variables may shadow options and lexically preceding (non-static) variables.)

You can't have two options with the same name, so that case isn't relevant.

The first part of that paragraph talks about "within the prolog" but then we say variables and options within the subpipeline can shadow each othere, except for the casual remark about not shadowing preceding static variables.

1. Why are we trying to prevent static variables from shadowing each other? What aspect of lexical scoping are we worried about?

2. Why are we trying to prevent non-static variables from shadowing preceding static variables or options?

[xatapult]

I don't have an answer to this question but I remember a lively discussion about this. Something to do with nasty edge cases... Wasn't it you, @xml-project, who brought this up?

[xml-project]

@eriksiegel It does not ring a bell. I will look at my note this afternoon. May be I will find something.

[xml-project]

See #550 und (for the do not shadow rule) #506 Also see the minutes from our first day at Leipzig last year.

[xml-project]

Looking at my notes from Leipzig last year, the answer to (1) is pure pragmatic. If they can't be shadowed you do not have to build a scope to find out the last shadowing, which is the real value.

I do not think it is worth it to allow scoping: Static variables are constants and a goody for pipeline authors because they do not have to write / change the same value over and over again. All this can be done without scoping -> choose another name for the new variable.

Concerning (2) I have to think about it.

[xml-project]

@ndw said:

You can't have two options with the same name, so that case isn't relevant.

I think what we were talking about is something like this:

<p:declare-step xmlns:p="http://www.w3.org/ns/xproc" 
    xmlns:xs="http://www.w3.org/2001/XMLSchema" 
    xmlns:test="http://dummy" version="3.0">
    <p:output port="result" sequence="true"/>
(1) <p:option name="option" static="true"  />
    <p:declare-step type="test:test">
        <p:output port="result" />
(2)     <p:option name="option" static="true" />
        <p:identity>
            <p:with-input><doc>{$option}</doc></p:with-input>
        </p:identity>
    </p:declare-step>
    <test:test />
</p:declare-step>

I do not like our solution either. If static options are basically static variables coming from the outside (the processor), I would argue they can only be declared in the outer most p:declare-step, but that was not our consensus. We agreed to make no distinction between the outer and any inner pipeline. This was why we said, they could not shadow.

[ndw]

No, I was talking about two options in the same declare-step not being allowed.

[xml-project]

OK, like:

<p:declare-step>
  <p:option static="true" name="option" select="1" />
  <p:option static="true" name="option" select="2  />
</p:declare-step>

If the option is set from the outside, it is pointless, because @select is never used. If it is not set from the outside it also pointless, because (allowing shadowing) it always "2". What did I miss?

Btw: If in my previous example the two "option" are different, how is a processor supposed to know, which is which and to set the two supplied values to the right option?

[ndw]

The example above is what I meant, but note that this is flatly invalid: you can't have two options on the same step with the same name.

In the earlier example, where a static option is declared in a nested step, the same rules apply. The expression that initializes that option must be computed statically and can refer to statics that are already in-scope. (Or that's how I understood it, anyway.)

[ndw]

By "the same rules" I meant, the rule that it has to be resolved statically.

[xml-project]

Sorry, I lost track. I was trying to explain why

You can't have two options with the same name, so that case isn't relevant.

I try to give an example what this means (because static option to my reading establish only one scope since there is no way to set them).

Do you actually propose to make the minor goody of static options/variables such a big thing. As I said, to me they make sense in the outermost pipeline and shouldn't be declarable in nested steps.

[ndw]

I don't think we should make nested pipelines different from imported pipelines and surely statics have to be initialized in imported pipelines, no?

[xml-project]

I don't think we should make nested pipelines different from imported pipelines

I know that we disagree in this point and we decided to follow your opinion. But imho that can't be the reason to make static options/variables a monster.

surely statics have to be initialized in imported pipelines, no?

Yes, but that does not mean inner and outer pipelines have to be the same. Remember my argument was to allow static option/variable declaration in a p:library. So if a step to be imported relies on a static option/variable, they have to be put in a p:library.

But I think we should not open up this discussion again. If I remember right, the rule "do not shadow static options/variables" was a consequence of allowing their declaration "everywhere" and the need to find a simple solution. You might say you do not want a simple solution anymore. But then we should concentrate the discussion on this point and its consequences.

I am not sure I even understand the semantics of shadowing a static option. But may be this is just my problem.

[xml-project]

I think I found the reason in my notes, why static variables do not shadow: Suppose

<p:library>
  <p:variable static="true" name="var" select="1" />
  <p:variable static="true" name="var" select="$var +1" />

What is the value of the imported variable? We need to make a rule for this.

[xatapult]

Gentleman. I find it worrying that this discussion pops up again. I thought we made a decision on this?

[xml-project]

@eriksiegel So what is your proposal?

[xatapult]

Stick to what we discussed. No shadowing of static variables and options.

[xml-project]

Fine with me.

[ndw]

The spec is unclear and self-contradictory so I don't believe that the decision was adequately recorded.

Observation: we went to a lot of trouble to make sure that variables and options (henceforth "variables", but I mean both) behave in the way users expect. We allow them anywhere and we allow them to shadow.

Opinions follow.

Assertion: the principle of least surprise should apply. A user who understands how variables work shouldn't be surprised by the behavior of a variable just because it's static.

1. Dynamic variables must be allowed to shadow static ones. Consider a pipeline with a bunch of nested, shadowed variables. Deciding to make the top one static shouldn't break the entire pipeline.
2. In separate pipelines, it must be possible to shadow statics. Pipelines are self-contained. If I pick one up and drop it into a larger pipeline, it shouldn't suddenly break because of the static variables in the surrounding pipeline.
3. Given 1 and 2, there's no compelling reason (that I can think of) not to allow statics to shadow each other. And it satisfies the principle of least surprise.

At any given point where an expression occurs, there is an unambiguous set of in-scope variables. Those are the ones that are used to evaluate the expression. If the variable is static, then any reference to non-static variables (or the context node) is an error.

static $foo = 5
static $foo = $foo + 1
$bar = $foo + 2  // $bar = 8
$foo = $foo + 9  // $foo = 15
static $baz = $foo + 1 // ERROR: $foo is not static

Additional constraints:

1. It is an error to have two p:options with the same name in the same prologue.
2. It is an error to have two different p:variables with the same name at the top level in a p:library

I'm a little unclear on how we intended a p:option in a p:library to ever get a different value at runtime, I suppose we could imagine a global context in which initial static option values are known. We probably need to be explicit about what happens in this case:

Given lib.xpl:

<p:library>
  <p:option name="foo" static="true" value="'libdefault'"/>
</p:library>

And a pipeline:

<p:declare-step>
  <p:option name="foo" static="true" value="'pipelinedefault'"/>

  <p:declare-step>
    <p:import href="lib.xpl"/>
    // What is the value of $foo here?
    // Does it depend on whether or not I provided a 'runtimevalue' value at runtime?
   ...
...

I think a case could be made for any of the three possible values. If the 'runtimevalue' is provided, I expect the only useful answer is 'runtimevalue. If no runtime value is provided, I'm inclined to say that the value should be libdefault. But I'm also inclined to remove p:option from p:library so we don't have to answer this question.

Consider this library:

<p:library>
  <p:option name="foo" static="true" value="10"/>

  <p:declare-step>
    $foo
  </p:declare-step>

  <p:declare-step>
    $foo
  </p:declare-step>
</p:library>

I assume the idea here is that $foo can be set once and used in multiple pipelines, which is handy. How would that differ in practice from:

<p:library>
  <p:declare-step>
    <p:option name="foo" static="true" value="10"/>
  $foo
  </p:declare-step>

  <p:declare-step>
    <p:option name="foo" static="true" value="10"/>
    $foo
  </p:declare-step>
</p:library>

Note that the pipelines could not have any other inner declaration of $foo because the outer value would have been shadowed in that case.

And in any event, are we really sure that static values from the "outer context" at runtime flow through into pipelines nested in libraries nested in pipelines nested in libraries?

[xml-project]

Just a suggestion without expressing any opinion (really, believe me). Aligning with XSLT 3.0 may be an option because our audience know the rules.

[ndw]

11 July editorial call: propose replace statics entirely with p:param(qname,defaultvalue,[type]) that returns a runtime value or the default. Norm to draft a PR that implements this for review.

[ndw]

This removes static options entirely; static variables still exist but cannot be shadowed.

[ndw]

1. That proposal failed miserably so we're back where we started.
2. I don't feel like I'm persuading anyone that we should just let static variables obey normal scoping rules.
3. I'm sure we don't want to begin discussing what the "import precedence" of imported static variables is.

The only other solution that I can think of is that we say that there's a single, global pool containing (the names of) all static variables and options.

1. The first occurrence of a static variable or option defines the value for that name.
2. The initializing expression is ignored on any subsequent occurrence.
3. If a subsequent occurrence specifies an as attribute, it is a static error if the current value does not satisfy that type.
4. If an implementation allows users to specify the values for options, those values constitute the "first occurrence".

So:

Given library.xpl:

<p:library>
  <p:variable static="true" name="libvar" select="5"/>
  <p:variable static="true" name="libopt" select="6" as="xs:integer"/>
</p:library>

and a pipeline:

<p:declare-step …>
  <p:import href="library.xpl"/>
  <p:option name="libopt" static="true" select="7"/>
  <p:option name="myopt" static="true" select="'hello' as="xs:string"/>
  <p:option name="myvar" static="true" select="8"/>

  <p:declare-step type="ex:foo">
    <p:option name="myopt" static="true" select="'goodbye'" as="xs:NMTOKEN"/>
    <p:option name="myotheropt" static="true" select="9"/>
     …
  </p:declare-step>

  <ex:foo myopt='test'/>

</p:declare-step>

At compile time:

1. If I specify (through some implementation-defined API) that the value of libopt is 100, then it has that value everywhere and no initializing expression matters.
2. If I do not specify a value, the value in the library is the “first occurrence” and it has the value 6 everywhere.
3. If I specify a value for myopt, it has that value everywhere and the initialzing expression in the nested subpipeline is ignored.
4. If I specify a value for myopt that does not satisfy the constraints of xs:NMTOKEN, a static error will be raised when that declaration is encountered.
5. If I don’t specify a value for myopt, the value in the outer pipeline is the first occurrence and it has the value “hello” everywhere.
6. The attempt to pass a value for myopt in the call to ex:foo is a static error.
7. Presumably, if I can specify the value for libopt through some API, I can equally specify the value for myotheropt through the same API.

I observe that we could simplify things by removing static options entirely and saying that static variables occupy a single, global pool and that implementations may provide a way to specify the initial contents of that pool.

I still object to the constraint that non-static variables can't shadow static ones. I don't see any reason for this restriction and it has the potential to really bite users. Imagine that I have a working pipeline with no static variables or options. I import a third party library in order to use a step it declares and suddenly my pipeline has a static error because there's a name collision between a static variable in that library and a local variable in my pipeline! How is that reasonable?

[ndw]

Per 29 Aug 2019 editor's call:

1. A single, global namespace for statics.
2. Only a single assignment is allowed for any static.
3. Only static options, no static variables.

It is forbidden to shadow static options with dynamic options or dynamic variables.

[xatapult]

@ndw: This issue must not be closed yet: The prose in the spec must still be changed (as we've discussed, I'll give this a try)

[ndw]

My PR (that you approved) attempted to make those changes. What did I miss?

[xatapult]

@ndw: You missed nothing. I missed that you also did the textual changes. I supposed you only did the schema changes and I still had to do the textual ones (as I proposed), but you did both. Thanks! :smile: