Upgrade Enso to GraalVM for JDK 21

JaroslavTulach commented 1 year ago

There is new version of GraalVM for JDK21 since yesterday. Here is some information about Truffle Unchained. The upgrade is going to be tougher this time:

### Tasks
- [x] Change the GraalVM version check to non-fatal error - **done** in #7855
- [x] Avoid `boostrap` & copy of `truffle-api`: `./bin/java -d org.graalvm.truffle` seems to `exports com.oracle.truffle.api` & co.  - **done** in #7855
- [x] Update documentation that mentiones `sbt bootstrap`
- [x] `gu` is gone! Stop calling it... https://github.com/oracle/graal/issues/6855 -  - **done** in #7855
- [x] `option -Dtruffle.class.path.append option is deprecated` - we need to build `module-info.java` for the runtime
- [x] Enable GraalPy on Windows: - https://github.com/oracle/graalpython/issues/321
- [x] Fix a lot of `Inexhaustive_Pattern_Match.Error TRegexReadOnlyIntArray` in `test/Tests`

Follow-ups

8313
8294
8290
8280
8314
https://github.com/enso-org/enso/issues/8321

enso-bot[bot] commented 11 months ago

Pavel Marek reports a new STANDUP for today (2023-10-06):

Progress: - Introducing modules into our sbt builds

Trying to compile all the java sources
Debugging a problem with sbt / zinc problem with mixed compilation of java and scala - cannot compile module-info.java.
sbt update does not work
maybe update scala? It should be finished by 2023-10-09.

enso-bot[bot] commented 11 months ago

Pavel Marek reports a new STANDUP for today (2023-10-09):

Progress: - sbt has a bug - cannot use module-info.java in mixed projects. Luckily, we need just one module-info in pure Java project runtime-with-instruments.

Still trying to compile just runtime project. Some progress.
sbt mixes together --module-path and --class-path in compilation and that causes problems.
Out BuiltinMethodProcessor fails on NoClassDefFoundError because of it. It should be finished by 2023-10-09.

enso-bot[bot] commented 11 months ago

Pavel Marek reports a new 🔴 DELAY for yesterday (2023-10-10):

Summary: There is 11 days delay in implementation of the Upgrade Enso to GraalVM for JDK 21 (#7851) task. It will cause 11 days delay for the delivery of this weekly plan.

Delay Cause: The initial estimate was too optimistic. Moreover, I worked on different issues in the meantime.

enso-bot[bot] commented 11 months ago

Pavel Marek reports a new STANDUP for yesterday (2023-10-10):

Progress: - Struggling with compilation of runtime.

Still failing on IllegalAccessError in annotation processor, even though javac receives correct --add-exports argument It should be finished by 2023-10-20.

enso-bot[bot] commented 11 months ago

Pavel Marek reports a new STANDUP for today (2023-10-11):

Progress: - Removed jdk.compiler dependency #8027

Finally compiled runtime.
Trying to assembly fat jar with module-info. Thinking where to put module-info.java - in runtime or in runtime-with-instruments? It should be finished by 2023-10-20.

enso-bot[bot] commented 11 months ago

Pavel Marek reports a new STANDUP for today (2023-10-13):

Progress: - Still fixing the compilation of runtime and runtime-with-instruments.

The progress so far:
- Implement a workaround for module-info.java compilation in sbt's mixed projects.
- Add module-info.java to the runtime project.
- Implement a custom merge strategy for module-info.class files, so that in the Uber jar, we have only one module-info.class file from org.enso.runtime module
- Assemble engine-runner.
Everything be able to compile, but the running of runner.jar still fails.
- I will need to look into the correct javaOptions for that. It should be finished by 2023-10-20.

enso-bot[bot] commented 11 months ago

Pavel Marek reports a new STANDUP for yesterday (2023-10-16):

Progress: - Book club, mails, chat, ...

Still looking into JPMS It should be finished by 2023-10-20.

enso-bot[bot] commented 11 months ago

Pavel Marek reports a new STANDUP for today (2023-10-17):

Progress: - Seems like we will have to ditch fat jars

Looking into how we can ditch fat jars.
Getting inspiration from how graalpython and FastR are built and run (module-path, class-path, ...) It should be finished by 2023-10-20.

enso-bot[bot] commented 11 months ago

Pavel Marek reports a new STANDUP for yesterday (2023-10-19):

Progress: - Tweaking sbt to provide two fat Jars - runtime.jar and runner.jar. Those fat jars should be almost exactly the same as are now.

Experimenting with running it.
The fat jars need to be explicit modules
Explicit modules cannot have conflicting packages, otherwise JVM initialization fails. It should be finished by 2023-10-20.

enso-bot[bot] commented 11 months ago

Pavel Marek reports a new STANDUP for today (2023-10-20):

Progress: - Investigating how to assembly runner.jar fat jar that has exclusive packages to runtime.jar fat jar. It should be finished by 2023-10-20.

enso-bot[bot] commented 11 months ago

Pavel Marek reports a new 🔴 DELAY for today (2023-10-23):

Summary: There is 7 days delay in implementation of the Upgrade Enso to GraalVM for JDK 21 (#7851) task. It will cause 7 days delay for the delivery of this weekly plan.

Delay Cause: A lot of struggling with assembling fat jars and running them.

enso-bot[bot] commented 11 months ago

Pavel Marek reports a new STANDUP for today (2023-10-23):

Progress: - Consultation with Jaroslav: Unblocked the problem with runner.jar as fat jar.

runner.jar shall not be a module, moreover, it shall not be provided on class-path, neither on module-path
Let's provide only runtime.jar as fat jar that shall be an explicit module provided on the module-path and that shall provide all the Enso-related lagnagues (enso, epb) and instruments
There will be runner.jar that will be loaded via our custom class loader.
Trying to launch all this stuff - simple -h option works, but still cannot run a simple Enso script
- Currently is failing on "ServiceConfigurationError: org.sl4j.spi.SLF4JServiceProvider: module org.enso runtime does not declareuses`" It should be finished by 2023-10-27.

JaroslavTulach commented 11 months ago

A lot of struggling with assembling fat jars and running them.

Yes, the problems with fat JARs were hard to solve. Their occurrence was a kind of unknown unknown to me. I apologize for not being around sooner to help overcome the problems with more advanced ClassLoader usage. I hope you are unblocked now.

will cause 7 days delay

Go for it! Still there is still going to be quite a few know unknowns, so the deadline is tough. Good luck.

enso-bot[bot] commented 11 months ago

Pavel Marek reports a new STANDUP for today (2023-10-24):

Progress: - Book club.

Remove slf4j and logback from runtime.jar.
- We should put them into module-path and treat them as explicit modules.
Fixing buildEngineDistribution sbt command.
- We need to copy all the necessary Jar modules inside component directory.
buildEngineDistribution seems to be working - a simple enso -h works.
Trying to add graaljs modules, and copy them to the distribution It should be finished by 2023-10-27.

enso-bot[bot] commented 11 months ago

Pavel Marek reports a new STANDUP for today (2023-10-25):

Progress: - Filter away all the graal-related JAR modules from withing our generated polyglot directories in std libs

Some small refactorings to build.sbt.
Fixing runtime / test configuration - module path, --add-exports, etc., to Test/javaOptions.
- testOnly *VectorTest works now.
Simple script with simple IO.println works now. It should be finished by 2023-10-27.

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new STANDUP for today (2023-10-26):

Progress: - Trying to fix class loading issues in runtime / test.

Trying to run Enso with nothing on module-path and everything on class-path.
- Inspiration in graal-js and graalpy.
Disabling class-path isolation with -Dpolyglotimpl.DisableClassPathIsolation=true works, but is not desirable, even for tests.
- Because then, there is no Truffle runtime compilation at all.
Debugging polyglot engine initialization errors - some ClassCastException that seems to be caused by different class loaders in ReplInstrumentProvider?
- Maybe our MessageEndpoint initialization is wrong? It should be finished by 2023-10-27.

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new STANDUP for today (2023-10-27):

Progress: - Reviews

Fixing unit tests by adding everything on class-path and disabling class-path isolation.
Fixing ./run build script.
Other small fixes: #8168 and #8166
Starting to look into the native image build. It should be finished by 2023-10-27.

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new 🔴 DELAY for today (2023-10-30):

Summary: There is 7 days delay in implementation of the Upgrade Enso to GraalVM for JDK 21 (#7851) task. It will cause 7 days delay for the delivery of this weekly plan.

Unit tests almost work, locally. We need to fix Rust build scripts, and native image builds, and then check that everything works. Let's hope that one more week is sufficient.

Delay Cause: Let's delay the task for one more week, as discussed

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new STANDUP for today (2023-10-30):

Progress: - Fixing native-image build - struggling with sbt task definitions and their dependencies

Striving to provide the easiest and quickest possible solution
- Adding stuff on module-path when building native image does not work again, let's put everything on class-path
Similar to how we do that for unit tests. It should be finished by 2023-11-03.

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new STANDUP for today (2023-10-31):

Progress: - Native image build fixes

Fixing license review - ublock CI - a lot of manual work, writing some script? It should be finished by 2023-11-03.

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new STANDUP for today (2023-11-01):

Progress: - Another PR for private module checking disabling fix - #8140

There is no need to download any GraalVM runtime - removing this functionality from project-manager, runtime-version-manager, and co.
Fixing invocation of language server from project-manager.
- We will need to start the language server subprocess in a diametrally different way.
- Previously, it was enough to just run something like java -jar runner.jar <args>. It should be finished by 2023-11-03.

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new STANDUP for today (2023-11-02):

Progress: - Fixing language server launch from project-manager, dealing with various class loading issues.

I was able to finally start the language server after these semi-manual modifications:
- Add a lot of --add-exports options
- Copy reference.conf from old runner.jar to the new runtime.jar.
- Move package akka.actor.typed from new runner.jar to the new runtime.jar.
- Remove contants of JAR archives that intersect runner.jar and runtime.jar from runner.jar - make runner.jar as thin as possible. It should be finished by 2023-11-03.

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new STANDUP for today (2023-11-03):

Progress: - language server can be launched from project-manager

It boots and connects to a client
In 15 seconds generates 5 GB of logs - the configuration is in runner.jar and not in runtime.jar.
- Now fixing the configuration - runtime module sees different configuration, e.g., for logging, than runner.jar.
Playing with specifying class loaders for the logging service It should be finished by 2023-11-03.

JaroslavTulach commented 10 months ago

* In 15 seconds generates 5 GB of logs - the configuration is in `runner.jar` and not in `runtime.jar`.
* Now fixing the configuration - runtime module sees different configuration, e.g., for logging, than `runner.jar`.

Pavel, @hubertp shall know what resources need to be read to configure SLF4J in the runtime.jar properly. The IsolatedClassLoader shall:

share the org.sfl4j classes between module path and runner.jar
load each SFL4J API class only once into the JVM
there should be only one implementation of appenders - it should be in runner.jar
Make sure the configuration resources from runtime.jar are visible to runner.jar SLF4J implementation
e.g. override getResource and getResources in IsolatedClassLoader
to delegate to system class loader to read necessary SLF4J configuration resources

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new 🔴 DELAY for today (2023-11-06):

Summary: There is 7 days delay in implementation of the Upgrade Enso to GraalVM for JDK 21 (#7851) task. It will cause 7 days delay for the delivery of this weekly plan.

Fixing language-server invocation from project-manager was more difficult than expected.

Delay Cause: The agreed upon delay for additional week.

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new STANDUP for today (2023-11-06):

Progress: - Still fixing log configuration

When running engine distribution from the cmd line, the log configuration is applied correctly from runner.jar.
When launching language-server from project-manager, file appender configuration seems ok, but console appender configuration is set to default, which causes generation of GBs of log messages in first few seconds. It should be finished by 2023-11-10.

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new STANDUP for today (2023-11-07):

Progress: - It seems that the log configuration on my branch was correct for some time, the weird behavior (generation of 5GB of output) can be reproduced on develop.

Update license review tool reports.
Fixing native image building once more. It should be finished by 2023-11-10.

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new STANDUP for yesterday (2023-11-07):

Progress: - Fixed native image of engine-runner.

Fixing other tests in project-manager and language-server. It should be finished by 2023-11-10.

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new STANDUP for today (2023-11-09):

Progress: - Fixed absolute paths to LanguageHome in some language-server tests.

Took a lot of time to realize what is going wrong in these async tests.
- Fixing tests in project-manager, tests in language-server succeeds locally. It should be finished by 2023-11-10.

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new STANDUP for today (2023-11-10):

Progress: - Fixed all project-manager tests.

It took some time, project-manager used some hacks to install engine and create and open projects in it.
- Fixed build of launcher native image
- Fixed tests in bench-processor.
- Fixed runtime-version-manager tests.
- Create follow-up issue #8280 It should be finished by 2023-11-10.

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new 🔴 DELAY for today (2023-11-13):

Summary: There is 7 days delay in implementation of the Upgrade Enso to GraalVM for JDK 21 (#7851) task. It will cause 7 days delay for the delivery of this weekly plan.

I hope this is the very last delay report for this issue, let's merge it ASAP.

Delay Cause: Tests in project-manager and co. were more difficult to fix than expected, they contained hacks that allowed them to launch the whole language-server. A week delay is a pessimistic estimation, if there are no blockers, it should be merged sooner.

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new STANDUP for today (2023-11-13):

Progress: - Fixing test in runtime-version-manager-test, project-manager, etc.

Trying to merge it ASAP. It should be finished by 2023-11-17.

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new STANDUP for today (2023-11-14):

Progress: - Fixed bug in sbt-assembly - now we can use the newest sbt-assembly plugin version and no IOException will be thrown when assembling.

All unit tests are passing, now Enso tests are failing
Discussing Jaroslav's serialization framework
Fixing java.sql.Driver service loading - java.sql.DriverManager is a bit dump and uses platform class loader to load drivers, so we need to do service loading ourselves. It should be finished by 2023-11-17.

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new STANDUP for today (2023-11-15):

Progress: - Fixed bug in Regex - graalvm.regex changed its regex polyglot object representation (groups).

Added some missing modules to the component directory - fixes more Enso tests.
Added chromeinspector tool, does not work though - freezes after chrome devtools connects.
- Won't solve in this PR.
Meetings It should be finished by 2023-11-17.

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new STANDUP for today (2023-11-16):

Progress: - Fixing std-benchmarks annotation processor.

Update parent class loader for our org.enso.runtime.HostClassLoader.
Will have to fix the benchmark runner such that it runs with truffle-compiler on module-path as well.
- Created follow-up issues after the PR is merged (in the issue description). It should be finished by 2023-11-17.

enso-bot[bot] commented 10 months ago

Pavel Marek reports a new STANDUP for today (2023-11-17):

Progress: - Fixed rest of the problems, should be able to merge in the evening.

Working on enabling JVM assertions on the CI in the meantime. It should be finished by 2023-11-17.

enso-org / enso

Upgrade Enso to GraalVM for JDK 21 #7851

Follow-ups

8313

8294

8290

8280

8314