Provide engine-runner & language-server as a separate JPMS modules

[Akirathan]

Current suboptimal module architecture

runner.jar is currently a fat plain jar (no a JPMS module) located in built-distribution/component/runner/runner.jar. System module-path is set to built-distribution/component. runner.jar contains all the classes from language-server and their dependencies. It does not contain classes that are on module-path. The original motivation, implemented in https://github.com/enso-org/enso/pull/7991, was to make the transition to JPMS as quick and as simple as possible. The idea is roughly as follows:

• Entry point is in org.enso.runtime module in EngineRunnerBootLoader class.
• This class creates an IsolatedClassLoader that either loads classes from runner.jar, or delegates to system class loader.
  • Delegation to system class loader is decided based on ClassLoaderConstants.CLASS_DELEGATION_PATTERNS
• system class loader loads classes from module-path. IsolatedClassLoader loads classes from runner.jar, that is not on the module-path.

This design, however, lead to many ugly hacks:

• A custom IsolatedClassLoader
• Defining unnecessary dependencies in runtime's module-info
  • Some of these dependencies are, in fact, needed only by runner.jar, like all the helidon modules. But we need to put them inside runtime's module-info so that the system class loader can lookup classes from these modules.

Better module architecture

• runner.jar should be a modular jar that is on the system's module-path.
• There should be no need for IsolatedClassLoader.

  
  ### Tasks

• [ ] language-server is separate JPMS module

[JaroslavTulach]

Related:

• 10010

• 7082

runner.jar should be a modular jar that is on the system's module-path.

Rather than that runner.jar (or rather just the language server JAR) should be a module JAR in Standard/Visualization/polyglot/java module, loaded by its own ModuleLayer which automatically delegates to (mostly empty ModuleLayer representing the JDK - e.g. java.base but no Truffle APIs or language implementations, etc.).

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-08-15):

Progress: - Starting to migrate engine-runner to JPMS module.

• I have to transitivelly migrate all the dependant projects to JPMS modules.
• So far, pretty smooth:
  • Refactored few smallish projects to Java only. This makes the transition to JPMS much easier.
  • Got rid of some package name clashes - shuffle the dependencies a bit. It should be finished by 2024-08-20.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-08-16):

Progress: - Do not run enterprise engine and stdlib benchmarks #10829.

• One failed attempt to make JPMSPlugin more automatic - https://github.com/enso-org/enso/issues/10726#issuecomment-2293325930
• Simplify declarations of module-path in sbt build - #10836 It should be finished by 2024-08-20.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-08-19):

Progress: - Continue migration of our internal projects to JPMS modules.

• Struggling with mixed Java/Scala projects.
• Not only we have to exclude module-info.java from the compilation and compile it manually, but the real pain is to ensure that there are correct requires statements in the module descriptor for Scala sources.
  • Obviously, when compiling Scala sources, requires statements in the module descriptor are not checked.
  • The problem arises when compiling a project that has this problematic mixed project as dependency - the compiler searches for, e.g., scala.util.Try, and fails. The only way to fix it is to add requires scala.library to the module descriptor.
  • Experimenting with a simple command line utility that ensures all the classes in the given module can be loaded.
    • java --validate-modules does not help.
  • Merged QoL improvement for the JPMS plugin - #10836. It should be finished by 2024-08-20.

[JaroslavTulach]

• add requires scala.library to the module descriptor.

Adding a dependency on Scala core library sounds like the correct fix.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-08-20):

Progress: - Providing org.enso.scalalibs.wrapper - a meta project that includes all the problematic Scala libraries.

• This will be a single module with module-info.
  • Shuffling some tasks in JPMSPlugin to simplify the build process. It should be finished by 2024-08-20.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-08-21):

Progress: - Migrating more internal projects to JPMS modules.

• Ultimatelly, will remove the runtime-fat-jar workaround. It should be finished by 2024-08-27.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-08-22):

Progress: - Still migrating more projects to JPMS modules and fixing compilation.

• Trying to boot enso for the first time and fixing related problems. It should be finished by 2024-08-27.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-08-23):

Progress: - Found a way how to compile module-info.java along with other Java sources and not as a standalone source.

• This helps tremendously to correctly set the requirements in other module descriptors.
  • Started modularization of language-server.
• This will take some time, as it has a lot of Scala classes and creating module-info.java for those is error prone and manual. It should be finished by 2024-08-27.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-08-26):

Progress: - Wrappers for akka and zio libraries.

• On latest PR version, updateLibraryManifests and buildEngineDistribution already works!
  • Tomorrow, will deal with runtime-integration-tests/test and language-server/test. It should be finished by 2024-08-27.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-08-27):

Progress: - Some tests already run in runtime-integration-test.

• Still have to deal with stuff like TestLogProvider inside logging-service-logback/Test, etc.
  • Quick fix for std libs benchmarks in https://github.com/enso-org/enso/pull/10902 It should be finished by 2024-08-27.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-08-28):

Progress: - enso --run test/Base_Tests and enso --run test/Table_Tests already works!

• Still have not resolved merge conflicts with develop on purpose - I probably want to split this huge PR into multiple smaller ones.
• Dealing with problems that the test packages are not opened for reflection - we try to test calling Java static methods inside the test classes from Enso and that fails. It should be finished by 2024-09-03.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-08-29):

Progress: - runtime-integration-tests/test seems to work locally!

• Fixing language-server/test. It should be finished by 2024-09-03.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-08-30):

Progress: - Many language server tests already succeeds.

• Still need to finish compilation of runtime-benchmarks and std-benchmarks, but after language-server, that should not be so difficult. It should be finished by 2024-09-03.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-09-02):

Progress: - Fixing problem with non-functional DirectoryWatcher

• Turns out, Scala's case class constructor swallows exceptions.
  • There are still various timeout errors in language server tests which are hard to debug. It should be finished by 2024-09-03.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-09-03):

Progress: - language-server/test fixed.

• runtime-benchmarks/run fixed.
• Dealing with issues with the annotation processor in std-benchmarks/compile.
  • The annotation processor cannot initialize EPB language for some reason. It should be finished by 2024-09-03.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-09-04):

Progress: - Merged and fixed with develop

• Some progress with running from the annotation processor - getting parent class loader from the current module layer
  • But org.graalvm.polyglot.Context is loaded by two different class loaders - investigating.
• Splitted rename PR in https://github.com/enso-org/enso/pull/10974 It should be finished by 2024-09-11.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-09-05):

Progress: - Renaming PR almost green, failing on some network transients. Will merge tomorrow.

• Running Enso code (benchmark collector) from annotation processor:
  • frgaal uses a custom class loader which is inherited to the annotation processors.
  • Truffle creates its own module layer and when Enso code is invoked, it is possible that, e.g., org.graalvm.polyglot.Context is loaded twice.
  • Trying to avoid this situation by explicitly creating our own module layer inside the annotation processor.
  • Seems easier than hacking around the class loaders. It should be finished by 2024-09-11.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-09-06):

Progress: - Our benchmark annotation processor generates sources.

• But I cannot get JMH annotation processor to work - frgaal cannot find the JMH processor. It should be finished by 2024-09-11.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-09-09):

Progress: - Small interactive session with Jaroslav about annotation processor problems

• Fix compiler error for unresolved symbol ready to merge - https://github.com/enso-org/enso/pull/11008
• Trying to improve performance of ExecutionContext.hasContextEnabled - https://github.com/enso-org/enso/issues/10702 It should be finished by 2024-09-11.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-09-11):

Progress: - Finally fixed annotation processing

• Fixing native image build It should be finished by 2024-09-11.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for the provided date (2024-09-14):

Progress: - High level discussion about future of Engine/LS.

• Presenting some ideas on benchmarking framework. It should be finished by 2024-09-25.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-09-18):

Progress: - Marked the PR as ready for review

• Fixing tests in project-manager
• Fixing annotation processors, again. It should be finished by 2024-09-25.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-09-19):

Progress: - Fixing booting of language server, and project-manager tests

• Needed to migrate few more internal projects to JPMS modules It should be finished by 2024-09-25.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-09-20):

Progress: - std libraries CI job is already green!

• Continuing with other fixes It should be finished by 2024-09-25.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-09-23):

Progress: - Fixing remaining tests, help from Hubert

• Created Postgres DB for benchmarks, populating with current data. It should be finished by 2024-09-25.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-09-24):

Progress: - With help of Hubert and Jaroslav, got all the other tests passing, along with some nice sbt refactorings

• license verification tool is now confused - what we distribute is not what license verification tool reports.
  • But there is still scala-compiler dependency distributed. Will try to get rid of that. It should be finished by 2024-09-25.

[enso-bot[bot]]

Pavel Marek reports a new STANDUP for today (2024-09-25):

Progress: - Last nits and reviews

• QA on some projects discovered few missing dependencies in some module declarations
• License verification
• All tests are green, some transient failures. Let's merge ASAP. It should be finished by 2024-09-25.