Generic term (de)construction loses attachments.

[AZWN]

Describe the bug In the Stratego 2 runtime, generic term (de)construction looses attachments.

Project We (@Gohla and me) observed the bug when using Stratego 2 as a source dependency. The actual code performing the term manipulation was in a library.

Versions Eclipse: org.eclipse.platform.ide 4.16.0.I20200604-0540 Spoofax: org.metaborg.spoofax.eclipse 2.6.0.20210820-110929-master System: Mac OS X x86_64 10.16

To Reproduce (ATTENTION: untested due to wizard error. See additional context for full explanation.)

1. Create a new Spoofax project with Stratego 2 enabled.

2. Change the syntax definition to

   Start.Program = <<{ID " "}+>>

3. Create a new builder with the following definition:

   imports
   libspoofax/term/origin
   
   rules
   show-origin-line: (ast, _, _, _, _) -> ()
    with
      _ := <explode-top; origin-line; debug(!"line: ")> ast
   
   explode-top c#(t*){a*} -> Op(c, t*){a*}

   4. Create a new program foo bar in this language
   5. Select foo, and execute the builder on it

Observed behaviour For any execution of the builder, line: 0 is logged

Expected behaviour Because attachments should be preserved by rewrite rules such as explode-top, we expect line: 1 to be logged.

Additional context We originally observed this bug when using the Statix runtime from Stratego 2 in Spoofax 3. Due to the missing attachments, no message locations were available, leading to poor error reporting. We observed that the term that was passed to the Statix solver from the Stratego runtime already lost its attachments. We worked around the issue by wrapping the explosion strategy in origin-track-forced: https://github.com/metaborg/nabl/commit/25bbf77cc0233f1a330511ad028c31da30fe5fe3. This led us to think that generic deconstruction is involved.

[Apanatshka]

While I bet it's unexpected behaviour to you, I would consider this expected behaviour. Origin tracking is not magic, and in fact is extremely limited in Stratego. I did a short write-up about the idea of origin tracking and how much is available in Stratego a while back that still needs to be adapted into documentation. I mean to improve origin tracking at some point in the future. But I currently consider this low priority, high effort.

[Gohla]

Why do you consider this expected? I would expect that when there is a rule from term a to b, that b has origin a after the rule is applied. Also, it is unexpected because with the old compiler, the origins do seem to be preserved (although I cannot test this completely in Spoofax 3 because I only support Stratego 2).

Whether implementing what is expected is feasible to do, is another thing of course. If there is indeed some documentation on what can be expected from origin tracking, that would be good to have to set expectations.

[Apanatshka]

There is currently an internal document that provides information on what is expected behaviour in the current version of Stratego and what might be useful behaviour in the future. I think we should definitely adapt that document into official Stratego documentation, but it's currently not public. I find it extremely unlikely that the old compiler has different behaviour because origin tracking is not implemented in the compiler at all, it's a dynamic thing that is entirely implemented in the runtime.

[Apanatshka]

Origin tracking is now publicly documented here: https://www.spoofax.dev/background/stratego/origin-tracking/