Unable to find `venv` on the host file system

[jonathon-bell]

Version: org.graalvm.python:python-language:23.1.0 OS: Linux Rocky 9

I have successfully:

• created a python virtual environment on my host file system
• installed my own python package 'my_package' within this env.
• created a Maven project to build a JAR file that defines a class foo that makes use of the polyglot API and the python language subsystem

• loaded my JAR, instantiated my class Foo and invoked the following code:

  
  ...
  public static foo() {
  var c = Context.newBuilder("python")
                 .allowAllAccess(true)
                 .allowNativeAccess(true)
                 .allowIO(true)
                 .option("log.level","FINE")
                 .logHandler(System.out)
                 .option("python.Executable", "file:/home/jonathon/Work/azg/master/lib/py_modules/venv/bin/python3")
                 .build();
  
  c.eval("python", "import site");
  c.eval("python", "import my_package"); // <-- ModuleNotFoundError: No module named 'my_package
  ....

I wish to source a python script from within my Java code that imports 'my_package' from within the host's virtual environment, however a different environment appears to be being used.

[python::Python3Core] FINE: Initial locations: Language home: /home/jonathon/Work/azg/master/lib/jar SysPrefix: BaseSysPrefix: CoreHome: StdLibHome: CAPI: JNI library: Home candidate: /home/jonathon/Work/azg/master/lib/jar [python::Python3Core] FINE: Initial locations: Language home: /home/jonathon/Work/azg/master/lib/jar SysPrefix: BaseSysPrefix: CoreHome: StdLibHome: CAPI: JNI library: Home candidate: /home/jonathon/.cache/org.graalvm.polyglot/python/python-home/7d0c656f075ff2153b202d151a09908b6a6056ef [python::Python3Core] FINE: Updated locations: Language home: /home/jonathon/.cache/org.graalvm.polyglot/python/python-home/7d0c656f075ff2153b202d151a09908b6a6056ef SysPrefix: /home/jonathon/.cache/org.graalvm.polyglot/python/python-home/7d0c656f075ff2153b202d151a09908b6a6056ef BaseSysPrefix: /home/jonathon/.cache/org.graalvm.polyglot/python/python-home/7d0c656f075ff2153b202d151a09908b6a6056ef CoreHome: /home/jonathon/.cache/org.graalvm.polyglot/python/python-home/7d0c656f075ff2153b202d151a09908b6a6056ef/lib/graalpy23.1 StdLibHome: /home/jonathon/.cache/org.graalvm.polyglot/python/python-home/7d0c656f075ff2153b202d151a09908b6a6056ef/lib/python3.10 Executable: file:/home/jonathon/Work/azg/master/lib/py_modules/venv/bin/python3 CAPI: /home/jonathon/.cache/org.graalvm.polyglot/python/python-home/7d0c656f075ff2153b202d151a09908b6a6056ef/lib/graalpy23.1 JNI library: /home/jonathon/.cache/org.graalvm.polyglot/python/python-home/7d0c656f075ff2153b202d151a09908b6a6056ef/lib/graalpy23.1 [python::GraalPythonModuleBuiltins] FINE: Setting default stdio encoding to utf-8:surrogateescape [python::Python3Core] FINE: import '_frozen_importlib' # [python::Python3Core] FINE: import '_frozen_importlib_external' # [python::Python3Core] FINE: # installing zipimport hook [python::Python3Core] FINE: import 'zipimport' # [python::Python3Core] FINE: # installed zipimport hook [python::Python3Core] FINE: import 'builtins' # [python::Python3Core] FINE: import 'graalpython' # [python::Python3Core] FINE: import '_weakref' # [python::Python3Core] FINE: import 'unicodedata' # [python::Python3Core] FINE: import '_sre' # [python::Python3Core] FINE: import 'function' # [python::Python3Core] FINE: import '_sysconfig' # [python::Python3Core] FINE: import 'java' # [python::Python3Core] FINE: import 'pip_hook' # [python::Python3Core] FINE: import '_struct' # Exception in thread "Thread-1" java.lang.ExceptionInInitializerError Caused by: org.graalvm.polyglot.PolyglotException: ModuleNotFoundError: No module named 'my_package' at .(Unknown) at org.graalvm.polyglot.Context.eval(Context.java:428) at com.csi.pyudx.Foo.(library.java:84) at com.csi.pyudx.Pi.(library.java:133)

Notice that although I set the 'python.Executable' property on the builder to point to where I believe the virtual environment is located, the Graal Python library appears to be unpacking its own virtual environment into the directory '/home/jonathon/.cache/org.graalvm.polyglot/python/python-home/7d0c656f075ff2153b202d151a09908b6a6056ef' and using that instead.

[tomasstupka]

Hi,

the best way to start with graalpy 23.1.0 would be to generate a polyglot graalpy java maven application skeleton graalpy -m standalone polyglot_app --output-directory MyJavaApplication https://www.graalvm.org/latest/reference-manual/python/standalone-binaries/ have a look into pom.xmlon how to create and populate with packages a venv togeter with graalpy and maven

another thing is that with graalpy 24.0.0, there will be:

• a maven archetype to create such maven applications - graalpy-archetype-polyglot-app
• and a graalpy maven plugin which simplifies the whole venv config and handling.

[jonathon-bell]

Thank you, Tomas,

I have done as you suggested and am studying the generated maven project.

I see this includes code for a custom class 'VirtualFileSystem' that implements the 'org.graalvm.polyglot.io.FileSystem' interface, an instance of which is installed into the 'Context.Builder' at startup via the (undocumented?) method 'Context.Builder.filSystem'.

Some questions: 1) where do I find more complete documentation of the full set of options supported by a context builder for the Python language 2) I don't fully understand the design that this particular 'VirtualFileSystem' class is implementing. It appears to unpack the contents of the embedded 'venv' resource tree into a temp directory and then resolve file look up requests via this the temp dir, but I do not yet understand all of the details. Is an overview of the design it implements available? 3) my original request / problem description still stands: I wish to construct a 'venv' virtual environment within a well known directory on the host machine, where a server (written in Java) loads and instantiates one or more plugins (each written in java and using polyglot to invoke a python script embedded as a resource within the plugin) and thus wish to share a common python virtual environment amongst all loaded plugins. Can you offer guidance as to how I modify the 'VirtualFileSystem' class (embedded within my plugin) so that it can also resolve python module imports using such a host's shared virtual environment?

Regards,

Jonathon

[tomasstupka]

1.) see $graalpy --help:all for a complete list 2.) Resources (e.g. python sources, etc) are loaded directly, only in some cases are files extracted, like with native extensions as they can't be loaded from a jar. 3.) .option("python.Executable") should specify the path, without the "file:" prefix. Did not notice previously, sry for that. Perhaps you will also need to add .option("python.ForceImportSite", "true").

[jonathon-bell]

Thanks, Tomas, this was very helpful. Also note that the option: ` .option("python.SysPrefix", path/to/venv) on a builder that uses the default file system enabled me to locate an existing 'venv' on the host hard drive.