How can I install a Python module from Java?

[ajh123]

In my Java project (which uses graalvm polyglot) I want to use a Python script which depends on the requests library - so I want to install it using pip.

Not sure if this is right place to post this, however I've not found a place for polyglot

This is what I tried so far:

version 1

    public static void installModule(Context context, String module) {
        context.eval("python", "import os\n" // line 32
                + "import pip\n"
                + "def install(package):\n"
                + "    if hasattr(pip, 'main'):\n"
                + "        pip.main(['install', package])\n"
                + "    else:\n"
                + "        pip._internal.main(['install', package])\n"
                + "install('"+ module +"')");
    }

version 2

    public static void installModule(Context context, String module) {
        context.eval("python", "import subprocess\n" // line 32
                + "import sys\n"
                + "def install(package):\n"
                + "    subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", package])\n"
                + "install('"+ module +"')");
    }

In version 1 I got this error:

Exception in thread "main" ModuleNotFoundError: No module named 'pip'
    at <python> <module>(Unknown)
    at org.graalvm.polyglot.Context.eval(Context.java:428)
    at uk.minersonline.gns34j.PythonConnector.installModule(PythonConnector.java:32)
    at uk.minersonline.gns34j.PythonConnector.main(PythonConnector.java:16)

In version 2 I got this error:

Exception in thread "main" OSError: (5, I/O error)
    at <python> __init__(C:\Users\samro\AppData\Local\org.graalvm.polyglot\python\python-home\7d0c656f1dac055b3b202d151a09908bf97221ff\Lib\subprocess.py:1687-1880:64732-73792)
    at <python> call(Unknown)
    at <python> check_call(Unknown)
    at <python> install(Unknown)
    at <python> <module>(Unknown)
    at org.graalvm.polyglot.Context.eval(Context.java:428)
    at uk.minersonline.gns34j.PythonConnector.installModule(PythonConnector.java:32)
    at uk.minersonline.gns34j.PythonConnector.main(PythonConnector.java:16)

This is how I am calling both versions:

    public static void main(String[] args) {
        Context context = Context.newBuilder().allowAllAccess(true).build();
        installModule(context,"requests"); # line 16
    }

What I want to know is what is the best way to install a Python module from Java.

[timfel]

The best way is currently (for the 23.1 release) to start with a project generated from graalpy's distribution:

Download a graalpy release and run this:

graalpy -m standalone polyglot_app -o graalpy_java_app

It will generated a skeleton project for you, check the pom.xml for how to install packages and the Java code for how to include them.

With the upcoming release 24.0, we have a maven plugin. Then it'll just be

    <plugins>
      <plugin>
        <groupId>org.graalvm.python</groupId>
        <artifactId>graalpy-maven-plugin</artifactId>
        <version>24.0.0</version>
        <executions>
          <execution>
            <configuration>
              <!-- specify python packages as if used with pip -->
              <packages>
                <package>termcolor==2.2</package>
              </packages>
              <pythonHome>
                <!-- java-like regular expression what file paths should be included into venv/home;
                     default is all -->
                <includes>
                  <include>.*</include>
                </includes>
                <!-- java-like regular expression what file paths should be excluded from venv/home;
                     default is none -->
                <excludes>
                </excludes>
              </pythonHome>
            </configuration>
            <goals>
              <goal>process-graalpy-resources</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>

[timfel]

FYI: Both of these solutions install the package into the Java resources at build time

[ajh123]

Thanks, I will have a look later. Once the maven plugin is released how could I use it from Gradle?

[timfel]

Thanks, I will have a look later. Once the maven plugin is released how could I use it from Gradle?

Our Gradle plugin is still under development and didn't make it for this release. We are planning to have a dedicated Gradle plugin in autumn

[ajh123]

@timfel On Windows 10 when I tried to run: graalpy -m standalone polyglot_app -o graalpy_java_app

I got this error:

Mar 04, 2024 2:32:14 PM org.graalvm.shadowed.org.jline.utils.Log logr
WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)
Error processing line 7 of C:\Users\samro\AppData\Roaming\Python\Python310\site-packages\pywin32.pth:

  Traceback (most recent call last):
    File "D:\Program Files (x86)\graalpy-23.1.2-windows-amd64\Lib\site.py", line 186, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "C:\Users\samro\AppData\Roaming\Python\Python310\site-packages\win32\lib\pywin32_bootstrap.py", line 23, in <module>
      os.add_dll_directory(path)
    File "D:\Program Files (x86)\graalpy-23.1.2-windows-amd64\Lib\os.py", line 1118, in add_dll_directory
      cookie = nt._add_dll_directory(path)
  AttributeError: module 'nt' has no attribute '_add_dll_directory'

Remainder of file ignored

GrealPy version

graalpy --version
Mar 04, 2024 2:33:25 PM org.graalvm.shadowed.org.jline.utils.Log logr
WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)
GraalPy 3.10.8 (Oracle GraalVM Native 23.1.2)

After doing this I uninstalled my existing Python 3.10.2 and still get the same errors. GraalPy's "bin" directory is on my path.

[ajh123]

However it looks like the graalpy_java_app folder was still created with its contents, so its a very random error.

[timfel]

After doing this I uninstalled my existing Python 3.10.2 and still get the same errors. GraalPy's "bin" directory is on my path.

Uninstalling Python 3.10.2 will not remove globally installed packages, and GraalPy will pick them up. You should remove "C:\Users\samro\AppData\Roaming\Python\Python310\" before running graalpy

[ajh123]

I think I know where I am going now. Thanks for the help.

[timfel]

I think I know where I am going now. Thanks for the help.

You're welcome! I'd be interested to hear about what you're doing if it is sth you want to share later on :)

[ajh123]

I can send a link to my repository once I get it working

[wangxi761]

pretty good,it slove my problem. but I don't see any relevant steps in the document about install packge from embedded java. Do I need to provide a pr to improve the document ？

[ajh123]

@wangxi761 It would be nice if you do, it would solve the problem for many more people.

Lack of documentation was the main reason why I submitted this issue ...

[timfel]

Does the documentation in the Maven archetype help? https://github.com/oracle/graalpython/blob/master/docs/user/README.md#maven

[dboggs95]

Does the documentation in the Maven archetype help? https://github.com/oracle/graalpython/blob/master/docs/user/README.md#maven

No. They don't. I have the same problem. Those docs tell me how to generate a python project with maven, but they don't explain to me this part:

<!-- specify python packages as if used with pip -->
<packages>
     <package>termcolor==2.2</package>
</packages>

[timfel]

Does the documentation in the Maven archetype help? https://github.com/oracle/graalpython/blob/master/docs/user/README.md#maven

No. They don't. I have the same problem. Those docs tell me how to generate a python project with maven, but they don't explain to me this part:

<!-- specify python packages as if used with pip -->
<packages>
     <package>termcolor==2.2</package>
</packages>

These packages are installed and included into the resources of the Java application, our maven plugin takes care of that. Doing that by hand (say, if you want to use Gradle) is rather involved. Refering to your points in https://github.com/oracle/graal/issues/8632, without a Gradle plugin, you would have to:

Step 1: Install GraalPy as a standalone as per https://www.graalvm.org/latest/reference-manual/python/Python-Runtime/#installing-graalpy Step 2: Enable GraalPy in your Java project, see the Maven archetype for reference: https://github.com/oracle/graalpython/blob/23e9e152541f731edf5878c7489a8a6fcbc5bcf4/graalpython/graalpy-archetype-polyglot-app/src/main/resources/archetype-resources/pom.xml#L20 Step 3: Use the standalone Python distribution to create a venv and install packages into it, just like you would with any other Python: https://www.graalvm.org/latest/reference-manual/python/Python-Runtime/#installing-packages Step 4: In your Java code, set up the Python context to find the venv you created. This can be via finding the path through resources, using our VirtualFileSystem, hardcoding and absolut path, ... The relevant options are documented in the Maven archetype: https://github.com/oracle/graalpython/blob/23e9e152541f731edf5878c7489a8a6fcbc5bcf4/graalpython/graalpy-archetype-polyglot-app/src/main/resources/archetype-resources/src/main/java/GraalPy.java#L104 Step 5: Execute a py script using the Context API: https://www.graalvm.org/latest/reference-manual/python/ and (linked on that page) https://www.graalvm.org/latest/reference-manual/embed-languages/ Step 6: Execute an individual py method and get results using the Context API: https://www.graalvm.org/latest/reference-manual/embed-languages/ Step 7: Run on SVM like any other Java application, there's nothing special to look out for other than where you decided to put your resources. Step 8: Compile the application to native - again, for Python nothing to do here, to include Python packages, copy the Maven archetype and how a VirtualFileSystem is used there: https://github.com/oracle/graalpython/blob/23e9e152541f731edf5878c7489a8a6fcbc5bcf4/graalpython/graalpy-archetype-polyglot-app/src/main/resources/archetype-resources/src/main/java/GraalPy.java#L82 and https://github.com/oracle/graalpython/tree/release/graal-vm/24.0/graalpython/graalpy-archetype-polyglot-app/src/main/resources/archetype-resources/src/main/resources/META-INF/native-image

Before we had the Maven plugin, the steps were similar there: https://medium.com/graalvm/supercharge-your-java-apps-with-python-ec5d30634d18