Investigate hosting yjs server within GraalJS

[jdunkerley]

Gui2 decided to bet on y.js. In order to avoid back and forth sending of package between processes, let's investigate hosting of the y-websocket server in language server process itself.

Language server process is capable to execute Graal.js - compliant EcmaScript implementation on top of JVM. All that is needed is to polyfill the WebSocket server API.

Initial polyfill is available at https://github.com/JaroslavTulach/graaljs-websocket

[JaroslavTulach]

Loading y-websocket Server

There is a graalvm/bin/js command to launch Graal.js in the GraalVM. If not, try graal/bin/gu install js. I am able to use that support with:

$ git clone https://github.com/yjs/y-websocket.git
$ cd y-websocket
$ rm -rf *; git checkout -f 05b2f894d5237005b70a3d3ab01f040111371506

$ npm install
$ npm run dist

$ cat >fill.js
function clearTimeout() {
}
function clearInterval() {
}
^D
$ /graalvm/bin/js --experimental-options --js.commonjs-require --js.commonjs-require-cwd=. fill.js ./dist/y-websocket.cjs

e.g. Graal.js (without node.js support) can load y-websocket.cjs and its dependencies.

[JaroslavTulach]

Using WebPack.js to generate single JavaScript file to send to Graal.js - easier than trying to convince Graal.js to understand mixture of CommonJS and ES6 modules.

See the commit

I assume packing things via webpack is OK, @Frizi? A standard thing among JavaScript developers, right?

[JaroslavTulach]

A meeting discussing Yjs synchronization with Engine has just finished. Here are the meeting notes. Feel free to update them to be more accurate:

• Issue: versioning of AST between engine & IDE
• @Frizi: can IDE send the code for the y.js to run on the engine side?
• is the API going to be simpler if it is AST?
• @4e6: gradually move...
  • step no. 1: Add UUID to each AST element & use UUIDs in other protocols/places

Question: y.js server in JVM. Good or bad? Proceed?

• Pros: eliminates one element of the infrastructure
  • step no. 1: run current y.js server in JVM (almost) unchanged
• Cons: question of compatibility still remains
• Need batch changes without engine seeing "middle stages"

@4e6 prepares a PR to move the server into the JVM.

[JaroslavTulach]

The strategic vision document has just been written. It describes why this hosting of y.js server in Graal.js is an important enabler.

[enso-bot[bot]]

Dmitry Bushev reports a new STANDUP for yesterday (2024-05-06):

Progress: Implemented the Ydoc launcher. Added Ydoc into the main module of the language server. Started debugging the startup. It should be finished by 2024-05-11.

Next Day: Next day I will be working on the #7954 task. Continue working on the task

[enso-bot[bot]]

Dmitry Bushev reports a new STANDUP for yesterday (2024-05-07):

Progress: Debugging the ydoc-server/run startup. Implemented the test for the bundled jar. Fixed the access to the polyfill URL class. It should be finished by 2024-05-11.

Next Day: Next day I will be working on the #7954 task. Continue working on the task

[enso-bot[bot]]

Dmitry Bushev reports a new STANDUP for today (2024-05-08):

Progress: Finished fixing the startup of ydoc-server/run. Debugging the startup of the language server. Found an issue with the discovery of Helidon service providers. Debugging the app classloaders. It should be finished by 2024-05-11.

Next Day: Next day I will be working on the #7954 task. Continue working on the task

[enso-bot[bot]]

Dmitry Bushev reports a new STANDUP for yesterday (2024-05-09):

Progress: Fixed an issue with the discovery of Helidon service providers. Added ydoc as a dependency to the runner-fat-jar. Removed Helidon classes form the runner jar. It should be finished by 2024-05-11.

Next Day: Next day I will be working on the #7954 task. Continue working on the task

[enso-bot[bot]]

Dmitry Bushev reports a new STANDUP for today (2024-05-10):

Progress: Debugging the issue with the javascript. Looks like it was introduced by the change in JS code. The diff only shows the changes in the parse tree library. Discovered that the Ydoc receives only the initial index event, and not the project-related event. Continue the investigation. It should be finished by 2024-05-11.

Next Day: Next day I will be working on the #7954 task. Continue working on the task

[enso-bot[bot]]

Dmitry Bushev reports a new 🔴 DELAY for yesterday (2024-05-13):

Summary: There is 7 days delay in implementation of the Investigate hosting yjs server within GraalJS (#7954) task. It will cause 0 days delay for the delivery of this weekly plan.

Delay Cause: Finishing integrating the Ydoc with the Language Server. The module path is fixed. Only one issue with the startup is left to fix.

[enso-bot[bot]]

Dmitry Bushev reports a new STANDUP for yesterday (2024-05-13):

Progress: Continue debugging the issue with the javascript. Found that the failure happens when the connection with WebSocket is established. Found that the build uses the Helidon version without the patch. Fixed the build. It should be finished by 2024-05-18.

Next Day: Next day I will be working on the #7954 task. Continue working on the task

[enso-bot[bot]]

Dmitry Bushev reports a new STANDUP for today (2024-05-14):

Progress: Finishing the integration of Ydoc with the language server. Fixed the module path in tests. Implemented the Ydoc JS bundle resource generation in SBT. Updated the license report It should be finished by 2024-05-18.

Next Day: Next day I will be working on the #7954 task. Continue working on the task

[enso-bot[bot]]

Dmitry Bushev reports a new STANDUP for yesterday (2024-05-15):

Progress: Dealing with the CI issues. Fixed the license reporter logic to filter out invalid paths from JARs. Fixed the linting errors. The Windows CI check fails on npm install. Debugging the issue. It should be finished by 2024-05-18.

Next Day: Next day I will be working on the #7954 task. Continue working on the task

[enso-bot[bot]]

Dmitry Bushev reports a new STANDUP for yesterday (2024-05-16):

Progress: Continue dealing with the CI issues. Added a Python setup step because some npm dependency requires it during the npm install. Cleaned up the PR and marked it ready. It should be finished by 2024-05-18.

Next Day: Next day I will be working on the #7954 task. Continue working on the task

[enso-bot[bot]]

Dmitry Bushev reports a new 🔴 DELAY for yesterday (2024-05-20):

Summary: There is 7 days delay in implementation of the Investigate hosting yjs server within GraalJS (#7954) task. It will cause 0 days delay for the delivery of this weekly plan.

Delay Cause: Continue working on integrating Ydoc

[enso-bot[bot]]

Dmitry Bushev reports a new STANDUP for yesterday (2024-05-20):

Progress: Returned to the PR. Working on review comments. Updated SBT build and included cached npm install build step. Removed npm install step from the CI setup. Removed unnecessary steps from the CI setup. It should be finished by 2024-05-25.

Next Day: Next day I will be working on the #7954 task. Continue working on the task

[enso-bot[bot]]

Dmitry Bushev reports a new STANDUP for today (2024-05-21):

Progress: Continue working on finishing the PR. Added ability to customize Ydoc hostname and port. Debugging a CI issue on Windows. Was able to reproduce the startup issue on the local virtual machine. It should be finished by 2024-05-25.

Next Day: Next day I will be working on the #7954 task. Continue working on the task

[JaroslavTulach]

Everything is ready for consumption by @PabloBuchu & co.