Still struggling to grasp the single scope rule for static options

[xml-project]

Consider:

<p:declare-step>
  <p:declare-step type="step:one">
    <p:option name="opt" static="true" select="42" />
    <!-- ... -->
  </p:declare-step>
  <p:declare-step type="step:two">
    <p:variable name="var" select="$opt" /> <!-- (1) -->
  </p:declare-step>
  <p:variable name="var" select="$opt" />   <!-- (2) -->

The single scope rule first says that I am not allowed to declare another option/variable with the same name in the global scope.

But what about visibility: If there is one scope, it seems to imply to me, that (1) and (2) are allowed. In other words, I can use any static option in an XPath expression, provided the declaration of this option lexically precedes the use.

What did I miss? And if I did not miss a think, should we state this more explictly in the specs?

[xatapult]

I would say that in your example (1) is not allowed here because $opt is declared inside a sibling declare-step...

[xml-project]

Would tend to agree in some way. On the other hand, it is declared because

  <p:option name="opt" />

would raise an error because of the single scope rule.

[ndw]

I think we have agreement on the following points.

1. There's a single global scope for statics.
2. A static option that's a direct child of a p:library is in the scope of a pipeline that imports that library.
3. In general, eachp:declare-step defines a distinct scope.
4. Except for the case in point 2 above, shadowing is defined lexically.

The most aggressive interpretation of the single scope rule combined with the no shadowing statics rule would say that because you've defined $opt static somewhere, all other attempts to create a variable or option named $opt must fail because it exists in the global scope.

I think that has really unpleasant consequences for the language. Consider a complicated pipeline that imports a few libraries, that also import a few libraries, etc. and everything is working fine:

<p:declare-step xmlns:p="..." xmlns:ex="...">
  <p:import href="a.xpl"/> <!-- imports b.xpl, c.xpl, d.xpl, and e.xpl -->
  <p:import href="b.xpl"/> <!-- imports d.xpl and e.xpl -->
  <p:declare-step type="ex:one">
    <p:import href="c.xpl"/>
    ...
  </p:declare-step>

  <p:option name="seropt">
  ...
</p:declare-step>

Now I decide to make $seropt static because I want to use it in a p:output. Suppose there's a step declared in e.xpl that happens to have a non-static option named $seropt. When I make my option static, I've introduced a static error into that pipeline?! Beyond the complexity of implementing that, there's the complexity of explaining it to the user. Which declaration is incorrect?

The argument that you should put all statics into your own namespace so that this problem doesn't arise still holds, of course, except that we know users don't want/don't understand/won't do that so I'm not sure they'll thank us for burning them when they fail to do it.

Is it not possible to say that the normal scoping rules apply to non-static variables and options? Can we not say that it's an error if they have the same name as a static option if-and-only-if the static option is in scope (including rule 2 above)? How is that significantly more difficult to implement or understand?

(Why can't that just generally be true for statics like any oth...no, no, no, I didn't say that. Nevermind. Move along, nothing to see here.)

[xml-project]

@ndw I am not sure I understand the consequences of your comment to my question / example. Could you elaborate it some more please.

[ndw]

I'll try. Consider these documents:

a.xpl:

<p:library xmlns:p="http://www.w3.org/ns/xproc"
           xmlns:ex1="http://example.com/one"
           version="3.0">

  <p:import href="b.xpl"/>

  <p:declare-step type="ex1:foo">
    <p:input port="source"/>
    <p:output port="result"/>
    <p:identity/>
  </p:declare-step>

</p:library>

b.xpl:

<p:library xmlns:p="http://www.w3.org/ns/xproc"
           xmlns:ex2="http://example.com/two"
           version="3.0">

  <p:declare-step type="ex2:bar">
    <p:option name="seropt" select="()"/>
    <p:input port="source"/>
    <p:output port="result"/>
    <p:identity/>
  </p:declare-step>

</p:library>

and pipe.xpl:

<p:declare-step xmlns:p="http://www.w3.org/ns/xproc" version="3.0">
  <p:input port="source"/>
  <p:output port="result"/>
  <p:option name="seropt" select="map { 'method': 'xml' }"/>

  <p:import href="b.xpl"/>

  <p:identity/>
</p:declare-step>

As it stands, pipe.xpl is statically valid and runs just fine (unless I've introduced typos in this post). Now suppose I decide that I want to use $seropt in the p:output, so I change my pipeline thus:

<p:declare-step xmlns:p="http://www.w3.org/ns/xproc" version="3.0">
  <p:option static='true' name="seropt" select="map { 'method': 'xml' }"/>
  <p:input port="source"/>
  <p:output port="result" serialization="$seropt"/>

  <p:import href="b.xpl"/>

  <p:identity/>
</p:declare-step>

All seems fine, right? Not if we adopt the most aggressive interpretation of a single, global scope for statics and no shadowing. If you look in b.xpl, imported by a.xpl, imported into our pipeline, you'll find that ex2:bar declares an option named $seropt. That is now a static error because it "shadows" a static. The fact that you've long since parsed that file and moved on when you encounter the static declaration is an implementation issue; explaining the error to users, that's the real problem.

It seems to me that the normal, lexical scoping rules should apply. The $seropt option in ex2:bar is miles away from the static declaration, I don't understand how they could interact. That being said, it's possible that I don't understand when lexical scoping of statics is a problem.

What's your interpretation of this pipeline?

[xml-project]

Thanks. I think now I have a clue how your answer is related to my question. Let me see, if I got it right:

You would argue that my thesis, that in

<p:declare-step>
  <p:declare-step type="step:one">
    <p:option name="opt" static="true" select="42" />
    <!-- ... -->
  </p:declare-step>
  <p:declare-step type="step:two">
    <p:variable name="var" select="$opt" /> <!-- (1) -->
  </p:declare-step>
  <p:variable name="var" select="$opt" />   <!-- (2) -->

at (1) $opt is visible is a too aggressive interpretation of the rules. If the normal lexical scoping rules apply, for which you argue, then $opt is not visible in (1) [as @eriksiegel agrees on].

Did I get you right?

I completely agree with this, I am not arguing for anything, but try to understand the complete consequences of the single-scope rule. Leaving out imports this boils down to "do-not-shadow-statics", right?

[ndw]

I argue that we should allow (1) and (2). Erik argues that (1) should be forbidden, if I understand him correctly. So we're not finished with this yet.

[xml-project]

I am completely confused, sorry. If you argue for scoping, how can (1) be allowed? $opt is not in scope in step:two. What did I missed?

[gimsieke]

Maybe we shouldn’t call it a “single global scope“ for statics. The usual scoping rules apply to static options (plus static options that are immediate children of an imported p:library are visible in importing steps), but an attempt to shadow an option is forbidden. We should rather call it a “single global name pool” for static options.

So (1) and (2) are allowed since the scope of <p:option name="opt" static="true" select="42" /> is limited to step:one.

In effect, it is probably only forbidden to import two libraries that declare two static options with the same name.

As a consequence of “p:declare-step always establishes a new scope”, we cannot refer to static options of the outer step declaration in a nested step, and we need to import the library twice if we want to refer to the static option in the nested declaration:

<p:library><!-- mylib.xpl -->
  <p:option name="foo" select="42" static="true"/>
</p:library>
<p:declare-step type="step:outer">
  <p:import href="mylib.xpl"/>
  <p:variable name="bar" select="$foo * 2"/><!-- ok -->
  <p:declare-step type="step:inner">
    <!-- not ok, need to import mylib.xpl here, too -->
    <p:variable name="baz" select="$foo * 4"/>
  </p:declare-step>
</p:declare-step>

[xml-project]

@gimsieke said:

So (1) and (2) are allowed since the scope of is limited to step:one.

So, I miss something obvious. How can (1) be allowed when the scope of $opt is limited to step:one? Where does the value of $opt come from?

[gimsieke]

Ah yes, I somehow assumed that there was an imported library that in the outermost p:declare-step that defined $opt. Without this, $opt is neither known in (1) nor in (2).

[gimsieke]

But even if there were such an imported library in the outermost p:declare-step, $opt wouldn’t be known in (1). But it wouldn’t be forbidden to declare a new static/non-static option or a variable in step:two that is named opt. Because the other, imported $opt is out of scope. Unless we imported the same libary that declares opt statically also in step:two.

[gimsieke]

This is I think the consequence of applying @ndw’s 4 rules outlined in https://github.com/xproc/3.0-specification/issues/868#issuecomment-527456266

[xml-project]

I do not argue against these rules, but

• I fail to understand the consequences @ndw draws for my initial problem.
• I am not sure these rules are clearly stated in the current specs.

But in favour of the single scope rule for static options, they advantage would be, that they would be visible in any child step declared in the importing step. I think the following example should work:

<p:declare-step name="main">
  <p:option static="true" name="debug" select="true" />

  <p:declare-step type="inner:one">
    <!-- ... -->
    <p:store use-when="$debug" />
  </p:declare-step>
</p:declare-step>

Therefor $debug should be visible in all child steps of main. Shouldn't it?

[ndw]

Sorry, I missread that as a declaration for $opt. I agree that (1) is a reference to a variable not in scope.

[gimsieke]

I do not argue against these rules, but

* I fail to understand the consequences @ndw draws for my initial problem.

* I am not sure these rules are clearly stated in the current specs.

But in favour of the single scope rule for static options, they advantage would be, that they would be visible in any child step declared in the importing step. I think the following example should work:

<p:declare-step name="main">
  <p:option static="true" name="debug" select="true" />

  <p:declare-step type="inner:one">
    <!-- ... -->
    <p:store use-when="$debug" />
  </p:declare-step>
</p:declare-step>

Therefor $debug should be visible in all child steps of main. Shouldn't it?

That is the question. I read “In general, eachp:declare-step defines a distinct scope.” as saying: A static option that is declared or imported in the outer step is not in scope in the embedded step.

This is why I proposed to do away with “global scope for static options” because that can make you think that the declare-step boundary is for all scopes except for static options.

If we impose strict declare-step boundaries also for statics, it is probably sufficient to say that an option must not try to shadow another option with the same name in the same scope. We don’t need to say anything about global scope or global name pool.

If you want to centralize static options, put them into libraries. Then import the libraries in each declare-step, even if a containing declare-step already imports the same library (or declares a static option with the same name).

[xml-project]

But doesn't that make static options somewhat useless? Do you really write p:import.... over and over again, when you write debug rules with @use-when or @serialization at locals p:store etc.

The ruling proposed is a viable way, I agree. But it too strict for my taste.

[gimsieke]

While the utility of static option may decrease a bit, it makes the language design cleaner.

If we go this route, it will probably considered a good practice to maintain a library serialization.xpl, statics.xpl, myframework-serialization.xpl, transpect-statics.xpl, … with common settings for HTML, XML, … serialization, and maybe even for other parameter maps. The additional verbosity of importing this library in every (serializing) declare-step is bearable IMHO.

(If the processor offers implementation-defined ways to override the default static options, all the better. The processor needs to make users aware of the fact that the overrides apply to each static option with the same name in every context – unless the processor offers an implementation-defined way to specify which static option in which libraries or step declarations should be overridden…)

[xml-project]

What about this:

1. Static options from their own global scope. As a consequence, a static option declared on a step is visible in all steps which are its children.

2. The initial static option scope is made up by all public static options that are direct children of the imported libraries / steps.

3. It is an error if two libraries / steps export static options with the same QName.

4. The global-option-scope is lexically scoped, so an inner step may declare a static option with a QName used in the declaration of a static options of one of its ancestors.

5. No step may declare two static options with the same name.

6. Static option scope always takes precedence, so no step may declare a non-static option with a name used for a static option declared or imported by one of its ancestors.

[gimsieke]

1. Static options from their own global scope.
   As a consequence, a static option declared on a step is visible in all steps which are its children.

Plus (?): A static option that is an immediate child of a library that is imported by the step is also visible in all child declare-steps.

What about if a step imports another step? Will the static options of the other step also be in-scope for this step and all its nested declare-steps?

[xml-project]

Plus (?): A static option that is an immediate child of a library that is imported by the step is also visible in all child declare-steps.

I think this is covered by (2.), isn't it?

What about if a step imports another step? Will the static options of the other step also be in-scope for this step and all its nested declare-steps?

I think the question goes to an imported step imports another step, right? I would expect that the same rulings as for importing steps would apply.

[gimsieke]

Plus (?): A static option that is an immediate child of a library that is imported by the step is also visible in all child declare-steps.

I think this is covered by (2.), isn't it?

Indeed. How can I have overlooked this?

What about if a step imports another step? Will the static options of the other step also be in-scope for this step and all its nested declare-steps?

I think the question goes to an imported step imports another step, right? I would expect that the same rulings as for importing steps would apply.

I had this example in mind:

<p:declare-step type="my:other"><!-- URI: other.xpl -->
  <p:option name="foo" select="'bar'" static="true"/>
  [subpipeline]
</p:declare-step>
<p:declare-step type="my:importing"><!-- URI: other.xpl -->
  <p:import href="other.xpl"/>
  <p:variable name="baz" select="$foo"/><!-- is $foo in scope? -->
  [subpipeline]
</p:declare-step>

According to “The initial static option scope is made up by all public static options that are direct children of the imported libraries / steps”, $foo should be in scope in my:importing.

[xml-project]

According to “The initial static option scope is made up by all public static options that are direct children of the imported libraries / steps”, $foo should be in scope in my:importing.

Right. I think that is what one would expect, isn't it?

[xatapult]

All right. Here's my view on things, trying to summarize what I understood was meant above (?):

1. Let's first try to establish that imports are handled like, can be viewed as, includes (p:library stripped off). After this (virtual) include operation we have a single document to talk about. Going from there makes defining scope much easier IMHO.
2. From this, let's define in scope: The following static options are in scope, reasoning from a child element of any p:declare-step:
   1. All static options declared in the same p:declare-step, defined before the current element, but not the ones in child p:declare-steps.
   2. All static options defined in the direct parent p:declare-step, defined before the start of the current p:declare-step, but not the ones in any sibling p:declare-steps
   3. The previous rule applies recursively to the parent of the parent, etc.
3. A static option that's in scope according to the rules above can be:
   1. Used/referenced
   2. Never redefined/shadowed/re-declared (whatever term). Either by a static or non-static option/variable.

Does this help? To me it feels intuitive and clear and relatively easy to explain :grimacing:

[ndw]

12 Sep editorial call, we feel pretty comfortable with Erik's summary.

Achim proposes that imports shouldn't be recursive in the sense that if A imports B and C imports A, that doesn't mean that B is visible in C. Needs more thought.

[ndw]

This is not like xsl:import, but maybe things are different enough that that's ok.

[ndw]

1. It is a static error if there is more than one visible declaration for a static option.
2. They have the same scope as step types.