search

ninia / jep

Embed Python in Java
Other
1.3k stars 147 forks source link

Unable to start JEP in M1 arm64 machine #442

Closed carlosuc3m closed 1 year ago

carlosuc3m commented 1 year ago

Describe the problem Hello, I m trying to set up JEP in a M1 arm64. I have already checked the architectures of the JVM and Python and they are both ARM64.

However, i am not able to load JEP. I get the following error:

Exception in thread "main" java.lang.UnsatisfiedLinkError: Can't load library: /Users/Cgarcia/git/deep-icy/miniconda3-arm64/envs/jepEnv_05112022_220358/lib/python3.9/site-packages/jep/libjep.jnilib

This is the code I am using:

   public static void main(String[] args) {
        MainInterpreter.setJepLibraryPath("/Users/Cgarcia/git/deep-icy/miniconda3-arm64/envs/jepEnv_05112022_220358/lib/python3.9/site-packages/jep/libjep.jnilib");
        PyConfig pyConfig = new PyConfig();
        pyConfig.setPythonHome("/Users/Cgarcia/git/deep-icy/miniconda3-arm64/envs/jepEnv_05112022_220358");
        MainInterpreter.setInitParams(pyConfig);
        try (SharedInterpreter interp = new SharedInterpreter()){
            interp.exec("3 + 3" + System.lineSeparator());
            interp.exec("import numpy as np" + System.lineSeparator());
            FloatBuffer data = ByteBuffer.allocateDirect(6*4).asFloatBuffer();
            DirectNDArray<FloatBuffer> nd = new DirectNDArray<>(data, 6);
            interp.set("x", nd);
            interp.exec("x[1] = 700");
            // val will 700 since we set it in python
            float val = data.get(1);
            data.put(4, val + 100);
            // prints 800 since we set in java
            interp.exec("print(x[4])");
            System.out.print(false);
        }
    }

Environment (please complete the following information):

Regards,

Carlos

bsteffensmeier commented 1 year ago

Please ensure that your Java and Python are both using the same architecture, since you indicate your python is arm64 you also need to ensure your java is using arm64.

If that does not resolve your issues I recommend reviewing the discussion on #379, #389, and #423.

carlosuc3m commented 1 year ago

I was using JAva 17 which was compatible with arm64 and it would not work. However, I also tried with Azul Zulu for Java 11 and Java 17 and got the same error. Is there any particular I could be ,issing for the installation of JEP? I did not build JEP from source (I installed it with pip) and I did not do anything else before trying to run it with Java. regards, Carlos

carlosuc3m commented 1 year ago

Hello again, I built JEP from source and now it works. However, I just did python setup.py build and not python setup.py install as install fails with:

Reading https://pypi.org/simple/jep/
/Users/Cgarcia/git/deep-icy/miniconda3-arm64/envs/aux/lib/python3.9/site-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning:  is an invalid version and will not be supported in a future release
  warnings.warn(
Downloading https://files.pythonhosted.org/packages/62/2e/3321cae2f4af21f9e4c32cce8f173b28a4385ebdc3ebc6525bbda6ad1adb/jep-4.1.0.tar.gz#sha256=c95a840350c4126a6cbe227dcb5cedbdd0a9467894cf00466a0a8c00c1f63c66
Best match: jep 4.1.0
Processing jep-4.1.0.tar.gz
Writing /var/folders/x5/8mtx0kxx0j74tdlqdn6l5p_00000gq/T/easy_install-1alox15y/jep-4.1.0/setup.cfg
Running jep-4.1.0/setup.py -q bdist_egg --dist-dir /var/folders/x5/8mtx0kxx0j74tdlqdn6l5p_00000gq/T/easy_install-1alox15y/jep-4.1.0/egg-dist-tmp-x9_hjdit
numpy include found at /Users/Cgarcia/git/deep-icy/miniconda3-arm64/envs/aux/lib/python3.9/site-packages/numpy/core/include
Traceback (most recent call last):
  File "/Users/Cgarcia/git/deep-icy/miniconda3-arm64/envs/aux/lib/python3.9/site-packages/setuptools/sandbox.py", line 156, in save_modules
    yield saved
  File "/Users/Cgarcia/git/deep-icy/miniconda3-arm64/envs/aux/lib/python3.9/site-packages/setuptools/sandbox.py", line 198, in setup_context
    yield
  File "/Users/Cgarcia/git/deep-icy/miniconda3-arm64/envs/aux/lib/python3.9/site-packages/setuptools/sandbox.py", line 259, in run_setup
    _execfile(setup_script, ns)
  File "/Users/Cgarcia/git/deep-icy/miniconda3-arm64/envs/aux/lib/python3.9/site-packages/setuptools/sandbox.py", line 46, in _execfile
    exec(code, globals, locals)
  File "/var/folders/x5/8mtx0kxx0j74tdlqdn6l5p_00000gq/T/easy_install-1alox15y/jep-4.1.0/setup.py", line 77, in <module>
    setup(name='jep',
  File "/Users/Cgarcia/git/deep-icy/miniconda3-arm64/envs/aux/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/Users/Cgarcia/git/deep-icy/miniconda3-arm64/envs/aux/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/Users/Cgarcia/git/deep-icy/miniconda3-arm64/envs/aux/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/Users/Cgarcia/git/deep-icy/miniconda3-arm64/envs/aux/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
    self.run_command(cmd)
  File "/Users/Cgarcia/git/deep-icy/miniconda3-arm64/envs/aux/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "/Users/Cgarcia/git/deep-icy/miniconda3-arm64/envs/aux/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 985, in run_command
    cmd_obj = self.get_command_obj(command)
  File "/Users/Cgarcia/git/deep-icy/miniconda3-arm64/envs/aux/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 860, in get_command_obj
    cmd_obj = self.command_obj[command] = klass(self)
  File "/Users/Cgarcia/git/deep-icy/miniconda3-arm64/envs/aux/lib/python3.9/site-packages/setuptools/__init__.py", line 158, in __init__
    super().__init__(dist)
  File "/Users/Cgarcia/git/deep-icy/miniconda3-arm64/envs/aux/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 59, in __init__
    raise TypeError("dist must be a Distribution instance")
TypeError: dist must be a Distribution instance

Is this expected?

carlosuc3m commented 1 year ago

Also I get the following error in one of the JEP commands that I do not get with other chip distributions:

Exception in thread "main" jep.JepException: <class 'TypeError'>: 'DirectNDArray' object does not support item assignment
    at <string>.<module>(<string>:1)
    at jep.Jep.exec(Native Method)
    at jep.Jep.exec(Jep.java:339)

at line:

interp.exec("x[1] = 700");

This does not happen with other operating systems or chips

carlosuc3m commented 1 year ago

It seems that JEP is not able to send the info to the Python Interpreter and to create it there, thus all the variables look as if they where not iniitailised

carlosuc3m commented 1 year ago

Ok I just fixed the error. Building JEP from source makes JEP and numpy work. For that I donwloaded the JEP source of version 4.1.0 from here qnd then did:

path/to/python/executable/python -m pip install /path/to/source/jep/folder/jep-4.1.0

With this JEP is built perfectly and works with numpy and everything.

Regards, Carlos

carlosuc3m commented 1 year ago

The issue was caused because I was trying to install it using the JAva ProcessBuilder to call the needed terminal commands. It seems that this class is not able to use the arm64 chips so it uses Rosetta to emulate x86_64 chips, which caused the creation of native libraries error. REgards, CArlos