Expose the Python truffle interpreter in a Maven package

[wetneb]

I would be interested in using the Python interpreter developed here from a Java application, as a replacement for the Jython library that we currently use. However it does not seem that this use of the codebase is documented anywhere?

As a user, I would find it useful to have a Maven package for this Python interpreter, which would depend on the appropriate Truffle packages. This would make it a lot easier to run this Python interpreter on a standard JVM, inside an existing Java application.

If given the appropriate pointers I would be happy to contribute a pull request to document that use in the README of this repository.

[timfel]

We don't release maven packages at the moment, that's true. More importantly, we have not tried to be compatible to Jython embedding at all, so most things work differently. I'm in the process of documenting how differently, I will update this issue once I can say more

[wetneb]

Thanks! Compatibility with the existing Jython interface would not be important for us - we cannot expect it to be a drop-in replacement. We would just need to know what is correct entry point to use GraalPython as a standalone library.

[timfel]

Can you share how you are using Jython? And are you aware that we are only targeting Python 3.7+, not Python 2.7 as Jython is, so there are incompatibilities on the language level that we won't be able to fix easily.

[timfel]

A good place to start to see the API to use Python from Java is here: https://www.graalvm.org/docs/reference-manual/embed/

[wetneb]

Yes we are aware of the differences between the Python versions - our aim is to get support for Python 3. We currently use Jython in https://github.com/OpenRefine/OpenRefine to propose Python as an expression language there (executing user-supplied scripts).

I can successfully call the Polyglot API from OpenRefine with the following Maven dependencies:

    <dependency>
        <groupId>org.graalvm.sdk</groupId>
        <artifactId>graal-sdk</artifactId>
        <version>19.1.1</version>
    </dependency>
    <dependency>
        <groupId>org.graalvm.truffle</groupId>
        <artifactId>truffle-api</artifactId>
        <version>19.1.1</version>
    </dependency>

But this does not seem to pull in the Python interpreter since calling context.eval("python", "3+8") raises java.lang.IllegalArgumentException: A language with id 'python' is not installed. Installed languages are: [].

[wetneb]

Just in case my message above is not clear: I would like to have Maven packages like graaljs does, but for graalpython, so that GraalPython can be used in any Maven project running on a stock JVM, as demonstrated here for GraalJS: https://github.com/graalvm/graal-js-jdk11-maven-demo

I am not sure how much work this entails, but if I attempted to do a PR for this, would it be something you would consider merging, @timfel? I would try to replicate whatever config is used in graaljs here (but of course this might require some work from the maintainers to publish the releases on Maven - hopefully the work can be minimal if the workflow is streamlined).

[wetneb]

After a quick investigation, it looks like some graalpython dependencies would also need to be packaged for Maven, for instance the Sulong API. It is not clear to me whether this is doable since Sulong contains platform-dependent code, if I understand things correctly.

[rmannibucau]

Seems it does not miss a lot, likely graalpython, sulong-api and sulong jars, if they are released on central, graalpython should be usable on another java 8 JVM, here is a pom doing that based on a local graalvm:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="
            http://maven.apache.org/POM/4.0.0
            http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.rmannibucau.github.test</groupId>
  <artifactId>test-python-graal</artifactId>
  <version>1.0-SNAPSHOT</version>

  <properties>
    <graal.base>${user.home}/.sdkman/candidates/java/19.3.0.r8-grl/jre</graal.base>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.graalvm.truffle</groupId>
      <artifactId>truffle-api</artifactId>
      <version>19.3.0.2</version>
    </dependency>
    <dependency>
      <groupId>org.graalvm.sdk</groupId>
      <artifactId>graal-sdk</artifactId>
      <version>19.3.0.2</version>
    </dependency>

    <dependency>
      <groupId>com.oracle.graal</groupId>
      <artifactId>graalpython</artifactId>
      <version>19.3.0</version>
      <scope>system</scope>
      <systemPath>${graal.base}/languages/python/graalpython.jar</systemPath>
    </dependency>
    <dependency>
      <groupId>com.oracle.graal</groupId>
      <artifactId>sulong-api</artifactId>
      <version>19.3.0</version>
      <scope>system</scope>
      <systemPath>${graal.base}/languages/llvm/sulong-api.jar</systemPath>
    </dependency>
    <dependency>
      <groupId>com.oracle.graal</groupId>
      <artifactId>sulong</artifactId>
      <version>19.3.0</version>
      <scope>system</scope>
      <systemPath>${graal.base}/languages/llvm/sulong.jar</systemPath>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.8.1</version>
        <configuration>
          <source>8</source>
          <target>8</target>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

[wetneb]

Yes, I have used a similar setup locally too. Although we can ship the jars in a local repository, it would be preferable to have them on Maven, so I have opened an issue for Sulong: https://github.com/oracle/graal/issues/2048

[Hi-Fi]

I think https://github.com/bytedeco/javacpp-presets has some way to solve that cross-os binary issue.

[timfel]

One thing to consider: when you use Graal Python, you should be running on GraalVM with Python installed (using gu) anyway. There's no public API in Python that you can use (there's literally zero stability guarantees for any API inside the com.oracle.graal.python package). Also, when running on HotSpot you will easily be 10x - 100x slower than CPython or Jython. We only optimize for running on GraalVM, so the best setup really is to use a GraalVM with Python installed to run your code, and then you don't need any Maven packages

[wetneb]

I understand, but here is my situation:

• In OpenRefine we cannot require our users to use GraalVM only - we also need to be able to run on a stock JVM if that is the only JVM that is available;
• Even if we required people to use the GraalVM, that would mean migrating out of Maven as a build system to use mx instead: that is a pretty big cost for us, and there is some risk associated to it too, since mx is not very widespread;
• The speed of execution of the Python interpreter is not a big issue anyway - those scripts are tiny and rarely computationally expensive.

Therefore I would still be interested in using GraalPython in a stock JVM.

[steve-s]

Even if we required people to use the GraalVM, that would mean migrating out of Maven as a build system to use mx instead: that is a pretty big cost for us, and there is some risk associated to it too, since mx is not very widespread;

Wouldn't it be enough just to set your JDK to GraalVM and then run Maven as usual? Note that API wise, you will only depend on the Graal SDK jar, which is published on Maven central (see https://github.com/oracle/graal/tree/master/sdk). If you execute something like myContext.eval("R", "1+3") on stock JVM, it will just tell you that language "R" is not available.

[timfel]

@wetneb like @steve-s said, you can still use Maven and depend on the GraalVM SDK, there's no reason to use mx.

[wetneb]

Ok, but then I would still need to somehow install the language evaluators via some external process as this will not be taken care of by the Maven build process… Do you see what I mean?

As a developer I want my users to be able to compile and run my project by calling a single build system which will fetch all dependencies for them, and I do not see how that can be done in a clean way here. Perhaps we could have a Maven plugin which would wrap the Graal language installer and specify in its configurations the languages we want to install, but even then I don't see how that could work nicely with packaged versions, since the language implementations would end up being installed on the developer's machine only and would not be included in distributions, right?

Or we can give up on distributing the language implementations in OpenRefine and install them dynamically at runtime, but that means the software is no longer usable offline by default, and the software will basically finish installing when first run… Do we agree that this is not very clean?

[wetneb]

Perhaps I just do not understand how you expect GraalPython to be used. As far as I can tell, at the moment, users need to:

• Install GraalVM
• Use gu to install GraalPython, since it is not included by default in the GraalVM CE distributions

This means that any Java project which wants to rely on GraalPython must require these two steps as a prerequisite to their own install process. Is this correct? If so, I am sure you can understand that we would lose a large part of our user base if we start requiring these steps, since the install process would become very complicated (requiring a stock JVM is already a big hurdle for many actually!)

But I totally understand if you are not interested in streamlining that sort of reuse because the project is still at an early stage, or if I misunderstood the scope and goals of the project.

I do think however that a large proportion of the projects which use Jython could eventually be interested in switching to GraalPython, given that Python 2.7 has reached its end of life, and GraalPython is the only Java implementation of Python 3 that I am aware of (Jython 3 is inactive).

I suspect OpenRefine is not the only project where Jython is used as an interpreter for small user-supplied scripts, where performance is not a big concern, and compatibility with C modules is not required. My understanding was that GraalPython could potentially fill this gap, but maybe not!

[eregon]

@wetneb What if the ./refine script would download GraalVM automatically + gu install python, and use that as the JDK? That could even mean users would no longer need to install a stock JDK manually.

[rmannibucau]

Agree GraaVM should deploy it on central as well, otherwise everybody will just deploy it on custom nexus or some artifactory which would create a lot of forks for no gain, for users nor for Graal project itself IMHO. The deployment is just a PUT to do during or after the release on oss.sonatype so no real reason to not do it for an OSS project.

[wetneb]

@eregon It would be possible, but that's not ideal, because requiring a large download during the first launch is not great (OpenRefine should be usable offline). Also the refine script is only used on Linux - for Windows and Mac we bundle OpenRefine differently so we would need to create custom launchers to install GraalVM in the same way (currently we are just using off-the-shelf launchers from launch4j and appbundler).

We do already bundle a JRE in our Mac package (because detecting Java on Mac is apparently harder with this tool) and there is also interest in a Windows version with a bundled JRE, which I hope to add soon - so in principle we could do the same with GraalVM (if I understand correctly that is allowed by the license), but that would generate really big distributions as GraalVM is really heavier than OpenJDK. And having to package software with embedded JREs is always a bit sad, no?

By the way, by the same token we should also require that any Spark cluster we connect to from OpenRefine runs on top of GraalVM as well, right? As a user of a managed Spark cluster I do not control what JVM it runs on… So I don't think it is realistic for us to expect that our workflows are always going to be executed on GraalVM. In 10 years maybe, if GraalVM has taken off as a widespread VM that users might already have on their machine because of other software. But today I don't feel like I can decide to impose it to users.

[timfel]

@rmannibucau except it is not just a PUT. There's native dependencies and the fact that it won't run properly on non-GraalVM, neither in terms of performance, nor in terms of sandboxing and security. So just uploading it somewhere will leave you with an inferior and silently slightly broken product.

[rmannibucau]

@timfel native deps can easily be bundled (it is what sigar does for instance), the sandboxing and security can be handled through a SecurityManager for part of it and can just be logged for the rest so still sounds like a good solution (the least bad? ;)). That said, happy if there is a more modular solution in the near future as well.

[wetneb]

I am curious about the sandboxing and security downsides of running GraalPython on a HotSpot JVM, where can I read more about that?

[chumer]

@wetneb did you have a look at: https://github.com/raydac/mvn-jlink Sounds like something that could be adopted to use for GraalVM CE downloads from Github.

[wetneb]

@chumer nice! But typically, it seems non-trivial to use mvn-jlink to solve this issue: although it could be used to download GraalVM as part of the build process, an extra step would be needed to gu install python on top of it, right?

[timfel]

@wetneb sorry, missed your question about sandboxing. I'm not sure if we have good resources about that (@chumer would know better), but on GraalVM you get classloader isolation for languages and @rmannibucau, note that we don't rely on the SecurityManager, the Truffle API provides sandboxing and isolation features that are tuned for the needs of languages (like more fine-grained limitations on IO).

[rmannibucau]

@timfel my point was more than a degraded mode using the SM sounds trivial and an acceptable compromise, the granularity would be bigger I guess but it is sufficient for JVM != Graal IMHO.

[eregon]

@wetneb Replying to some of your points,

@eregon It would be possible, but that's not ideal, because requiring a large download during the first launch is not great (OpenRefine should be usable offline).

Doesn't the script execute mvn anyway on first launch? (which requires an internet connection)

We do already bundle a JRE in our Mac package (because detecting Java on Mac is apparently harder with this tool) and there is also interest in a Windows version with a bundled JRE, which I hope to add soon - so in principle we could do the same with GraalVM (if I understand correctly that is allowed by the license), but that would generate really big distributions as GraalVM is really heavier than OpenJDK. And having to package software with embedded JREs is always a bit sad, no?

GraalVM enables capabilities that simply are not available on stock JREs (e.g., JITing code of guest languages), so I think it would make sense to ship as a dependency. As you noticed, it's already a hot mess to find a JDK in a cross-platform way, so I think installing it as a dependency would make sense and simplify a lot. As an example, IntelliJ comes with its own JDK.

Re GraalVM is large, that's true, maybe we should provide a smaller base tarball, comparable in size to OpenJDK, and then just gu install python for your case?

[wetneb]

Doesn't the script execute mvn anyway on first launch? (which requires an internet connection)

It only does so if the software has not been built yet (if you are checking out the project from git for instance). If you try out our Linux packaged versions you will see that Maven is not used at all and nothing is downloaded.

GraalVM enables capabilities that simply are not available on stock JREs (e.g., JITing code of guest languages), so I think it would make sense to ship as a dependency. As you noticed, it's already a hot mess to find a JDK in a cross-platform way, so I think installing it as a dependency would make sense and simplify a lot. As an example, IntelliJ comes with its own JDK.

Re GraalVM is large, that's true, maybe we should provide a smaller base tarball, comparable in size to OpenJDK, and then just gu install python for your case?

Say you manage to halve the size of GraalVM releases: it is still going to be a hard sell for us. I would not double/triple the size of our packaged releases for the sake of migrating from Python 2 to Python 3, given the current role of Python in the software.

Adding GraalVM as a dependency would also be a lot of packaging/maintenance work. I would rather spend that time packaging GraalPython for Maven if it can benefit others (it would still be heavier than Jython, but to a reasonable extent). Sorry, I am a bit stubborn!

[timfel]

@wetneb just to re-iterate: running GraalPython on OpenJDK is very slow even compared to Jython or CPython. For example, it takes 10s to launch a Context if you want to have local packages available, ~2-3 minutes to import NumPy on my machine ... we are not optimizing for that use case and in fact we're not even testing it in the CI right now. --- my point being that I don't think a Maven packaging would benefit anyone, it will only confuse people when they cannot do anything meaningful with it.

[rmannibucau]

I guess this ticket must be taken as "let's replace jython in apps with graal python" and open the impl to become more mainstream.

[wetneb]

@timfel I have already done small tests and performance looks acceptable so far for the uses cases we typically deal with.

[shoaniki]

One problem with "gu install python" is that the gu command doesn't actually exist in graalvm distributions for Windows.

[laiweiwei]

One problem with "gu install python" is that the gu command doesn't actually exist in graalvm distributions for Windows.

.\gu.cmd --help

[thadguidry]

@eregon Circling back to compromise :-) We recently in OpenRefine begin publishing a OpenRefine with embedded JRE. Knowing where @wetneb has concerns, I'm curious as you mentioned earlier if you might be able to begin to provide a small GraalVM dedicated to only Python and Java? This can help get others in our OpenRefine community interested in our Python support to lend a hand to @wetneb (our primary developer currently) and help look at packaging a OpenRefine-GraalVM-Python release for our OpenRefine users who might be interested? That way perhaps everyone wins, and we might alleviate @wetneb concerns of size and maintenance on OpenRefine's end? How does that sound to everyone?

[eregon]

@thadguidry Currently there is only 1 GraalVM base image (and it contains JavaScript, etc) as part of the release assets.

However, it's fairly easy to build your own GraalVM with a list of components to include: https://github.com/oracle/graal/blob/master/vm/README.md

So for the case of just GraalPython on JVM (no native images) and with the GraalVM compiler to JIT Python code, here is an env file:

DYNAMIC_IMPORTS=graalpython,/compiler
COMPONENTS=Graal.Python,GraalVM compiler

Then you can build it with:

• Install mx
• Download the corresponding JVMCI JDK, see https://github.com/oracle/graal/blob/vm-20.1.0/common.json from https://github.com/graalvm/graal-jvmci-8/releases or https://github.com/graalvm/labs-openjdk-11 and set JAVA_HOME to the extracted path

  
  git clone --branch vm-20.1.0 https://github.com/oracle/graal.git
  cd graal/vm

Save the env file above to e.g. mx.vm/python-jvm-ce

Check the included components (will also clone graalpython automatically)

mx --env python-jvm-ce graalvm-show

mx --env python-jvm-ce build

The result is in

mx --env python-jvm-ce graalvm-home

[thadguidry]

@eregon Thanks! Let me see how this works and feels.

[thadguidry]

@eregon How does this look for the final 3 checks for included components? By the way, I used the release/graal-vm/20.2 branch https://github.com/oracle/graal/tree/release/graal-vm/20.2

E:\graal>cd graal/vm

E:\graal\graal\vm>mx --env python-jvm-ce graalvm-show
Cloning https://github.com/graalvm/graalpython.git revision 9fe7c233a380d2db462894cfa3f6ce3f8fe5d39a to E:\graal\graalpython with Git
Cloning into 'E:\graal\graalpython'...
remote: Enumerating objects: 3343, done.
remote: Counting objects: 100% (3343/3343), done.
remote: Compressing objects: 100% (1323/1323), done.
Receiving objects: 100% (130429/130429), 76.84 MiB | 22.92 MiB/s, done.d 127086

Resolving deltas: 100% (81408/81408), done.
Updating files: 100% (4803/4803), done.
Downloading LLVM_ORG_SRC from ['https://lafo.ssw.uni-linz.ac.at/pub/llvm-org/llvm-src-llvmorg-9.0.0-5-g80b1d876fd-bg0c808efbe5.tar.gz']
 116291993 bytes (100%)
GraalVM distribution: GRAALVM_DB59B483DC_JAVA11
Version: 20.2.0-dev
Config name: None
Components:
 - Graal.Python ('pyn', /python)
 - GraalVM compiler ('cmp', /graal)
 - Graal.Python license files ('pynl', /python)
 - Truffle ('tfl', /truffle)
 - Sulong ('slg', /llvm)
 - LLVM.org toolchain ('llp', /llvm)
 - TRegex ('rgx', /regex)
 - Graal SDK ('sdk', /graalvm)
 - Truffle NFI ('nfi', /nfi)
Launchers:
 - graalpython.cmd (bash)
 - lli.cmd (bash)
Missing exe suffix: graalvm-native-clang

Then...

E:\graal\graal\vm>mx --env python-jvm-ce build
JAVA_HOME: E:\Downloads\labsjdk-ce-11.0.8-jvmci-20.2-b03
28 unsatisfied dependencies were removed from build (use -v to list them)
23 non-default dependencies were removed from build (use -v to list them, mx build --all to build them)
Compiling com.oracle.mxtool.compilerserver with javac(JDK 11)... [E:\graal\mx\mxbuild\java\com.oracle.mxtool.compilerserver\bin\com\oracle\mxtool\compilerserver\CompilerDaemon.class does not exist]
Compiling com.oracle.mxtool.checkcopy with javac-daemon(JDK 11)... [E:\graal\mx\mxbuild\java\com.oracle.mxtool.checkcopy\bin\com\oracle\mxtool\checkcopy\CheckCopyright.class does not exist]
Downloading ASM_7.1 from ['https://repo1.maven.org/maven2/org/ow2/asm/asm/7.1/asm-7.1.jar', 'https://search.maven.org/remotecontent?filepath=org/ow2/asm/asm/7.1/asm-7.1.jar']
 114762 bytes (100%)
Downloading sources ASM_7.1 from ['https://repo1.maven.org/maven2/org/ow2/asm/asm/7.1/asm-7.1-sources.jar', 'https://search.maven.org/remotecontent?filepath=org/ow2/asm/asm/7.1/asm-7.1-sources.jar']
 172521 bytes (100%)
Downloading ASM_TREE_7.1 from ['https://repo1.maven.org/maven2/org/ow2/asm/asm-tree/7.1/asm-tree-7.1.jar', 'https://search.maven.org/remotecontent?filepath=org/ow2/asm/asm-tree/7.1/asm-tree-7.1.jar']
 50303 bytes (100%)
Downloading sources ASM_TREE_7.1 from ['https://repo1.maven.org/maven2/org/ow2/asm/asm-tree/7.1/asm-tree-7.1-sources.jar', 'https://search.maven.org/remotecontent?filepath=org/ow2/asm/asm-tree/7.1/asm-tree-7.1-sources.jar']
 73045 bytes (100%)
Downloading ASM_ANALYSIS_7.1 from ['https://repo1.maven.org/maven2/org/ow2/asm/asm-analysis/7.1/asm-analysis-7.1.jar', 'https://search.maven.org/remotecontent?filepath=org/ow2/asm/asm-analysis/7.1/asm-analysis-7.1.jar']
 33379 bytes (100%)
Downloading sources ASM_ANALYSIS_7.1 from ['https://repo1.maven.org/maven2/org/ow2/asm/asm-analysis/7.1/asm-analysis-7.1-sources.jar', 'https://search.maven.org/remotecontent?filepath=org/ow2/asm/asm-analysis/7.1/asm-analysis-7.1-sources.jar']
 40510 bytes (100%)
Downloading ASM_COMMONS_7.1 from ['https://repo1.maven.org/maven2/org/ow2/asm/asm-commons/7.1/asm-commons-7.1.jar', 'https://search.maven.org/remotecontent?filepath=org/ow2/asm/asm-commons/7.1/asm-commons-7.1.jar']
 70394 bytes (100%)
Downloading sources ASM_COMMONS_7.1 from ['https://repo1.maven.org/maven2/org/ow2/asm/asm-commons/7.1/asm-commons-7.1-sources.jar', 'https://search.maven.org/remotecontent?filepath=org/ow2/asm/asm-commons/7.1/asm-commons-7.1-sources.jar']
 78696 bytes (100%)
Downloading JACOCOCORE_0.8.4 from ['https://repo1.maven.org/maven2/org/jacoco/org.jacoco.core/0.8.4/org.jacoco.core-0.8.4.jar', 'https://search.maven.org/remotecontent?filepath=org/jacoco/org.jacoco.core/0.8.4/org.jacoco.core-0.8.4.jar']
 193727 bytes (100%)
Downloading sources JACOCOCORE_0.8.4 from ['https://repo1.maven.org/maven2/org/jacoco/org.jacoco.core/0.8.4/org.jacoco.core-0.8.4-sources.jar', 'https://search.maven.org/remotecontent?filepath=org/jacoco/org.jacoco.core/0.8.4/org.jacoco.core-0.8.4-sources.jar']
 160840 bytes (100%)
Downloading JACOCOREPORT_0.8.4 from ['https://repo1.maven.org/maven2/org/jacoco/org.jacoco.report/0.8.4/org.jacoco.report-0.8.4.jar', 'https://search.maven.org/remotecontent?filepath=org/jacoco/org.jacoco.report/0.8.4/org.jacoco.report-0.8.4.jar']
 128699 bytes (100%)
Downloading sources JACOCOREPORT_0.8.4 from ['https://repo1.maven.org/maven2/org/jacoco/org.jacoco.report/0.8.4/org.jacoco.report-0.8.4-sources.jar', 'https://search.maven.org/remotecontent?filepath=org/jacoco/org.jacoco.report/0.8.4/org.jacoco.report-0.8.4-sources.jar']
 112319 bytes (100%)
Downloading JOPTSIMPLE_4_6 from ['https://repo1.maven.org/maven2/net/sf/jopt-simple/jopt-simple/4.6/jopt-simple-4.6.jar', 'https://search.maven.org/remotecontent?filepath=net/sf/jopt-simple/jopt-simple/4.6/jopt-simple-4.6.jar']
 62477 bytes (100%)
Downloading sources JOPTSIMPLE_4_6 from ['https://repo1.maven.org/maven2/net/sf/jopt-simple/jopt-simple/4.6/jopt-simple-4.6-sources.jar', 'https://search.maven.org/remotecontent?filepath=net/sf/jopt-simple/jopt-simple/4.6/jopt-simple-4.6-sources.jar']
 74419 bytes (100%)
Compiling com.oracle.mxtool.jacoco with javac-daemon(JDK 11)... [dependency JACOCOREPORT_0.8.4 updated]
Downloading JMH_GENERATOR_ANNPROCESS_1_21 from ['https://repo1.maven.org/maven2/org/openjdk/jmh/jmh-generator-annprocess/1.21/jmh-generator-annprocess-1.21.jar', 'https://search.maven.org/remotecontent?filepath=org/openjdk/jmh/jmh-generator-annprocess/1.21/jmh-generator-annprocess-1.21.jar']
 30848 bytes (100%)
Downloading sources JMH_GENERATOR_ANNPROCESS_1_21 from ['https://repo1.maven.org/maven2/org/openjdk/jmh/jmh-generator-annprocess/1.21/jmh-generator-annprocess-1.21-sources.jar', 'https://search.maven.org/remotecontent?filepath=org/openjdk/jmh/jmh-generator-annprocess/1.21/jmh-generator-annprocess-1.21-sources.jar']
 29146 bytes (100%)
Downloading COMMONS_MATH3_3_2 from ['https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.2/commons-math3-3.2.jar', 'https://search.maven.org/remotecontent?filepath=org/apache/commons/commons-math3/3.2/commons-math3-3.2.jar']
 1692782 bytes (100%)
Downloading sources COMMONS_MATH3_3_2 from ['https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.2/commons-math3-3.2-sources.jar', 'https://search.maven.org/remotecontent?filepath=org/apache/commons/commons-math3/3.2/commons-math3-3.2-sources.jar']
 2005074 bytes (100%)
Downloading JMH_1_21 from ['https://repo1.maven.org/maven2/org/openjdk/jmh/jmh-core/1.21/jmh-core-1.21.jar', 'https://search.maven.org/remotecontent?filepath=org/openjdk/jmh/jmh-core/1.21/jmh-core-1.21.jar']
 512901 bytes (100%)
Downloading sources JMH_1_21 from ['https://repo1.maven.org/maven2/org/openjdk/jmh/jmh-core/1.21/jmh-core-1.21-sources.jar', 'https://search.maven.org/remotecontent?filepath=org/openjdk/jmh/jmh-core/1.21/jmh-core-1.21-sources.jar']
 400038 bytes (100%)
Compiling com.oracle.mxtool.jmh_1_21 with javac-daemon(JDK 11)... [dependency JMH_1_21 updated]
Downloading HAMCREST from ['https://repo1.maven.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar', 'https://search.maven.org/remotecontent?filepath=org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar']
 45024 bytes (100%)
Downloading sources HAMCREST from ['https://repo1.maven.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3-sources.jar', 'https://search.maven.org/remotecontent?filepath=org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3-sources.jar']
 32624 bytes (100%)
Downloading JUNIT from ['https://repo1.maven.org/maven2/junit/junit/4.12/junit-4.12.jar', 'https://search.maven.org/remotecontent?filepath=junit/junit/4.12/junit-4.12.jar']
 314932 bytes (100%)
Downloading sources JUNIT from ['https://repo1.maven.org/maven2/junit/junit/4.12/junit-4.12-sources.jar', 'https://search.maven.org/remotecontent?filepath=junit/junit/4.12/junit-4.12-sources.jar']
 200355 bytes (100%)
Compiling com.oracle.mxtool.junit with javac-daemon(JDK 11)... [dependency JUNIT updated]
Note: E:\graal\mx\java\ListModules.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Compiling com.oracle.mxtool.junit.jdk9 with javac-daemon(JDK 11)... [E:\graal\mx\mxbuild\java\com.oracle.mxtool.junit.jdk9\bin\com\oracle\mxtool\junit\ModuleSupport.class does not exist]
Compiling com.oracle.mxtool.webserver with javac-daemon(JDK 11)... [E:\graal\mx\mxbuild\java\com.oracle.mxtool.webserver\bin\com\oracle\mxtool\webserver\WebServer.class does not exist]
Compiling org.graalvm.collections with javac-daemon(JDK 11)... [E:\graal\graal\sdk\mxbuild\src\org.graalvm.collections\bin\org\graalvm\collections\EconomicMap.class does not exist]
Compiling org.graalvm.collections.test with javac-daemon(JDK 11)... [dependency JUNIT updated]
Compiling org.graalvm.word with javac-daemon(JDK 11)... [E:\graal\graal\sdk\mxbuild\src\org.graalvm.word\bin\org\graalvm\word\ComparableWord.class does not exist]
Compiling org.graalvm.options with javac-daemon(JDK 11)... [E:\graal\graal\sdk\mxbuild\src\org.graalvm.options\bin\org\graalvm\options\OptionCategory.class does not exist]
Compiling org.graalvm.nativeimage with javac-daemon(JDK 11)... [dependency org.graalvm.word updated]
Compiling org.graalvm.home with javac-daemon(JDK 11)... [dependency org.graalvm.nativeimage updated]
Compiling org.graalvm.home.test with javac-daemon(JDK 11)... [dependency JUNIT updated]
Compiling org.graalvm.polyglot with javac-daemon(JDK 11)... [dependency org.graalvm.collections updated]
Downloading JLINE from ['https://repo1.maven.org/maven2/jline/jline/2.14.6/jline-2.14.6.jar', 'https://search.maven.org/remotecontent?filepath=jline/jline/2.14.6/jline-2.14.6.jar']
 268780 bytes (100%)
Compiling org.graalvm.launcher with javac-daemon(JDK 11)... [dependency org.graalvm.polyglot updated]
Compiling org.graalvm.launcher.test with javac-daemon(JDK 11)... [dependency JUNIT updated]
Compiling org.graalvm.nativeimage.test with javac-daemon(JDK 11)... [dependency JUNIT updated]
Compiling org.graalvm.polyglot.tck with javac-daemon(JDK 11)... [dependency org.graalvm.polyglot updated]
Building graalpython.exe.image-bash... [E:\graal\graal\sdk\mxbuild\windows-amd64\graalpython.exe.image-bash\graalpython.cmd does not exist]
Building lli.exe.image-bash... [E:\graal\graal\sdk\mxbuild\windows-amd64\lli.exe.image-bash\lli.cmd does not exist]
Building graalvm-native-clang.image-bash... [E:\graal\graal\sdk\mxbuild\windows-amd64\graalvm-native-clang.image-bash\graalvm-native-clang.cmd does not exist]
Missing exe suffix: graalvm-native-clang

E:\graal\graal\vm>

Then...

E:\graal\graal\vm>mx --env python-jvm-ce graalvm-home
E:\graal\graal\sdk\mxbuild\windows-amd64\GRAALVM_DB59B483DC_JAVA11\graalvm-db59b483dc-java11-20.2.0-dev

E:\graal\graal\vm>

Is that last output for mx --env python-jvm-ce graalvm-home result correct? Because the real path in Windows Explorer (breadcrumb) looks like this:

[timfel]

@thadguidry neither Python nor Sulong work on Windows, yet. For macOS or Linux, your commands look good

[thadguidry]

@timfel Ah that's what I figured. Thanks!

[thadguidry]

@timfel Just to get clarity, I "could" build on WSL2 on Windows and then have a custom GraalVM that can be used on Windows, macOS, Linux, correct?

[timfel]

You can build on WSL2, but the native parts of the GraalVM build are platform specific, so it'll only run on on Linux. If you're only running in JVM mode and are not going to use C extensions it might work, but we don't test that scenario

[j-baker]

I found this issue when looking at a similar situation. At the company I work for, we sometimes use the Graal Javascript execution engine on a standard JVM (standard JVMs are mandated by the organization for reasons such as security scanning simplicity). As far as I know, it runs reasonably performantly (the argument against here), because we enable JVMCI and use --upgrade-module-path to enable the Graal compiler and so the JIT does work. From my reading, this leads to a similar performance situation to if you use a GraalVM implementation.

In such a model, adding support for JS is super easy - just depend on the one Maven coordinate and watch it go. Unfortunately, this only works for JS because that's the only Maven jar published. It would be pleasant if the same thing worked for Python.

[timfel]

@j-baker thanks for that use-case perspective. However, the argument is not only one about performance, but also native dependencies and other resources. Python needs a lot, and will only add more over time. JS is a pretty small and self-contained language outside the node-js ecosystem; Python only exists as a language + stdlib that exposes and assumes platform-specific low-level access to the system.

[Ecorous]

Is there any update on this? I'd really like to use GraalPython to replace Jython, and yes I am aware of the performance downsides

[timfel]

Is there any update on this? I'd really like to use GraalPython to replace Jython, and yes I am aware of the performance downsides

No updates, sorry. As I said further up, the issue is that someone would have to figure out how to ship the native dependencies for all platforms in the package, and then you will always have to allow access to tmpdir for extracting the stdlib, llvm interpreter, and our native extensions prior to startup. Doable, but quite some work and maintenance overhead.

[mehrat]

Any update on this ? We have a use case to use GraalPython with stock JDK.

[chumer]

If everything goes well you will be able to use GraalPy from the application module-path also on a stock JDK in the upcoming release 23.1.

More info here: https://github.com/oracle/graal/issues/6852

[chumer]

@mehrat We are currently gathering some feedback on this topic in our community chat: https://graalvm.slack.com/archives/CNBFR78F9/p1691241119810719 if you can find some spare minutes to answer my questions there, that would be great! Anyone can join the community slack here: https://www.graalvm.org/slack-invitation/

[wetneb]

This has indeed been solved in release 23.1. Thanks a ton for that!

See:

• https://medium.com/graalvm/truffle-unchained-13887b77b62c
• https://www.graalvm.org/latest/reference-manual/embed-languages/
• https://github.com/graalvm/polyglot-embedding-demo