Deliver expression updates via Ydoc AST elements

[JaroslavTulach]

To proof the concept of delivering information from the engine to the IDE(s) via Ydoc structures is viable, let's prototype delivery of expression updates. To do following tasks are needed:

### Tasks
- [ ] Design changes to the Ydoc AST structures, so they contain [expression updates](https://github.com/enso-org/enso/blob/5afcb15ff2635f6701afaee2a3927f43fa4f26ed/docs/language-server/protocol-language-server.md#expressionupdate)
- [ ] Implement a way to launch a [GraalVM Insight](https://www.graalvm.org/latest/tools/graalvm-insight/) script in the `EnsoContext` created by `language-server` and expose Ydoc AST to it - see [manual steps](https://github.com/enso-org/enso/issues/11159#issuecomment-2379235662)
- [ ] Use Insight script to update the Ydoc structures with _expression update_ information
- [ ] Change the IDE to use this information instead of the one obtained via [language server protocol](https://github.com/enso-org/enso/blob/5afcb15ff2635f6701afaee2a3927f43fa4f26ed/docs/language-server/protocol-language-server.md)

If this proofs viable, then we can continue delivering visualizations via the AST mechanism.

[JaroslavTulach]

I was chatting about this with @Frizi and if I remember it correctly, the initial step is to setup a TypeScript project to use Vue core and Insight and generate a single file we can then feed into the --vm.D=polyglot.insight=my_script.js and see whether it obtains reactive events properly. I provided Pawel following notes:

• Generic info on GraalVM Insight: https://www.graalvm.org/latest/tools/graalvm-insight/
• Insight APIs (in Java form, no typescript): https://www.graalvm.org/tools/javadoc/org/graalvm/tools/insight/Insight.html
• An example that shows some dynamic reconfiguration triggered via HTTP connection: https://www.graalvm.org/latest/tools/graalvm-insight/tracing/

We used following sample Enso program:

from Standard.Base import all
import MyCustom.Library

fac n =
    facacc n v =
        if n <= 1 then v else
            @Tail_Call facacc n-1 n*v

    res = facacc n 1
    res

main n=10 k=5 =
    nf = fac n
    kf = fac k
    nkf = Warning.attach "a problem" fac n-k

    nf / (kf*nkf)

instrumented by following JavaScript (with embedded Enso) sketch:

let cache = new Map();

/*
let m = Polyglot.eval("enso", `
    from Standard.Base import all
    import MyCustom.Library

    main =
        6 * 5
`);
*/

insight.on("enter", (ctx, frame) => {
    print("About to execute: " + ctx.name + " at line " + ctx.line)
    for (let p in frame) {
        print(`  value of ${p} is ${frame[p]}`);
    }
    //
    cache.get("...")
    ctx.returnNow(5)
}, {
    //statements : true,
    // expressions : true,
    roots : true,
    // expressions : true,
})

insight.on("return", (ctx, frame) => {
    print(" finish: " + ctx.characters + " return value: "  + ctx.returnValue(frame))
}, {
    //statements : true,
    expressions : true,
    rootNameFilter : '.*main'
})

the whole sample was then executed as

enso --run ~/fac.enso --vm.D=polyglot.insight=c.js

GraalVM Insight

GraalVM is an advanced JDK with ahead-of-time Native Image compilation.

Insight

declaration: package: org.graalvm.tools.insight, class: Insight

Tracing with Insight

GraalVM is an advanced JDK with ahead-of-time Native Image compilation.

[JaroslavTulach]

Using GraalVM Insight in the IDE

There already is a way to use GraalVM Insight in the IDE. Works with 2024.5.1-nightly.2024.9.27. Here are the steps to follow: create a file x.py and put following code into it:

print("Starting GraalVM Insight!")

def onExit(ctx, frame):
    print(f"Exit from {ctx.name} at {ctx.charIndex}-{ctx.charEndIndex} with {ctx.returnValue(frame)}")

#class At:
#    sourcePath = ".*agent-fib.js"

class Expressions:
    expressions = True
    # at = At()
    # rootNameFilter = ".*main"

insight.on("return", onExit, Expressions())

into it. Then execute the project manager as:

enso$ ENSO_JVM_OPTS=-Dpolyglot.insight=`pwd`/x.py sbt runProjectManagerDistribution

open a project and observe the output. You may need following patch to make the output of print function visible:

diff --git engine/language-server/src/main/scala/org/enso/languageserver/io/ObservableOutputStream.scala engine/language-server/src/main/scala/org/enso/languageserver/io/ObservableOutputStream.scala
index 6cb12517ff..bc6caf8c17 100644
--- engine/language-server/src/main/scala/org/enso/languageserver/io/ObservableOutputStream.scala
+++ engine/language-server/src/main/scala/org/enso/languageserver/io/ObservableOutputStream.scala
@@ -18,12 +18,14 @@ class ObservableOutputStream extends OutputStream {
   /** @inheritdoc */
   @TruffleBoundary
   override def write(byte: Int): Unit = lock.synchronized {
+    System.err.write(byte)
     notify(Array[Byte](byte.toByte))
   }

   /** @inheritdoc */
   @TruffleBoundary
   override def write(bytes: Array[Byte]): Unit = lock.synchronized {
+    System.err.write(bytes)
     if (bytes.length > 0) {
       notify(bytes)
     }
@@ -33,6 +35,7 @@ class ObservableOutputStream extends OutputStream {
   @TruffleBoundary
   override def write(bytes: Array[Byte], off: Int, len: Int): Unit =
     lock.synchronized {
+      System.err.write(bytes, off, len)
       if (len > 0) {
         val buf = new Array[Byte](len)
         Array.copy(bytes, off, buf, 0, len)