Unable to install jep-4.2.0 on Python 3.6.13 with Java 11

[pvb05]

Describe the problem We are unable to install jep-4.2.0 on Python 3.6.13 (pyenv environment) and it throws errors when compiling the SubInterpreterOptions.java.

/usr/lib/jvm/java-11-amazon-corretto.x86_64/bin/javac -deprecation -d build/java -h build/include -classpath src src/main/java/jep/JepConfig.java src/main/java/jep/ClassList.java src/main/java/jep/Jep.java src/main/java/jep/Interpreter.java src/main/java/jep/LibraryLocator.java src/main/java/jep/DirectNDArray.java src/main/java/jep/SharedInterpreter.java src/main/java/jep/NamingConventionClassEnquirer.java src/main/java/jep/MainInterpreter.java src/main/java/jep/PyConfig.java src/main/java/jep/PyMethod.java src/main/java/jep/AbstractNDArray.java src/main/java/jep/SubInterpreterOptions.java src/main/java/jep/ClassEnquirer.java src/main/java/jep/SubInterpreter.java src/main/java/jep/NDArray.java src/main/java/jep/JepAccess.java src/main/java/jep/Run.java src/main/java/jep/Proxy.java src/main/java/jep/JepException.java src/main/java/jep/python/PyObject.java src/main/java/jep/python/MemoryManager.java src/main/java/jep/python/InvocationHandler.java src/main/java/jep/python/PyPointer.java src/main/java/jep/python/PyCallable.java src/main/java/jep/python/PyBuiltins.java
    src/main/java/jep/SubInterpreterOptions.java:61: error: unmappable character (0xE2) for encoding US-ASCII
         * allocator state. Otherwise it will use (share) the main interpreter???s.
                                                                              ^
    src/main/java/jep/SubInterpreterOptions.java:61: error: unmappable character (0x80) for encoding US-ASCII
         * allocator state. Otherwise it will use (share) the main interpreter???s.
                                                                               ^
    src/main/java/jep/SubInterpreterOptions.java:61: error: unmappable character (0x99) for encoding US-ASCII
         * allocator state. Otherwise it will use (share) the main interpreter???s.
                                                                                ^
    src/main/java/jep/SubInterpreterOptions.java:114: error: unmappable character (0xE2) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                    ^
    src/main/java/jep/SubInterpreterOptions.java:114: error: unmappable character (0x80) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                     ^
    src/main/java/jep/SubInterpreterOptions.java:114: error: unmappable character (0x99) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                      ^
    src/main/java/jep/SubInterpreterOptions.java:114: error: unmappable character (0xE2) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                                             ^
    src/main/java/jep/SubInterpreterOptions.java:114: error: unmappable character (0x80) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                                              ^
    src/main/java/jep/SubInterpreterOptions.java:114: error: unmappable character (0x99) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                                               ^
    src/main/java/jep/SubInterpreterOptions.java:128: error: unmappable character (0xE2) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                    ^
    src/main/java/jep/SubInterpreterOptions.java:128: error: unmappable character (0x80) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                     ^
    src/main/java/jep/SubInterpreterOptions.java:128: error: unmappable character (0x99) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                      ^
    src/main/java/jep/SubInterpreterOptions.java:128: error: unmappable character (0xE2) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                                             ^
    src/main/java/jep/SubInterpreterOptions.java:128: error: unmappable character (0x80) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                                              ^
    src/main/java/jep/SubInterpreterOptions.java:128: error: unmappable character (0x99) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                                               ^
    src/main/java/jep/SubInterpreterOptions.java:162: error: unmappable character (0xE2) for encoding US-ASCII
         * this is false use (share) the main interpreter???s GIL. When this is true
                                                         ^
    src/main/java/jep/SubInterpreterOptions.java:162: error: unmappable character (0x80) for encoding US-ASCII
         * this is false use (share) the main interpreter???s GIL. When this is true
                                                          ^
    src/main/java/jep/SubInterpreterOptions.java:162: error: unmappable character (0x99) for encoding US-ASCII
         * this is false use (share) the main interpreter???s GIL. When this is true
                                                           ^
    src/main/java/jep/SubInterpreterOptions.java:163: error: unmappable character (0xE2) for encoding US-ASCII
         * use the sub-interpreter???s own GIL.
                                  ^
    src/main/java/jep/SubInterpreterOptions.java:163: error: unmappable character (0x80) for encoding US-ASCII
         * use the sub-interpreter???s own GIL.
                                   ^
    src/main/java/jep/SubInterpreterOptions.java:163: error: unmappable character (0x99) for encoding US-ASCII
         * use the sub-interpreter???s own GIL.
                                    ^
    21 errors
    error: command '/usr/lib/jvm/java-11-amazon-corretto.x86_64/bin/javac' failed with exit status 1
    ----------------------------------------

Environment (please complete the following information):

• OS Platform, Distribution, and Version: Amazon Linux 2.0.20240109.0
• Python Distribution and Version: 3.6.13 built and installed via Pyenv with shared modules options.
• Java Distribution and Version: Java 11 - java-11-amazon-corretto-headless-1:11.0.22+7-1.amzn2.x86_64
• Jep Version: 4.2.0
• Python packages used (e.g. numpy, pandas, tensorflow): Numpy 1.19.5, Scipy 1.5.4

Logs

[INFO] DOCKER> Collecting numpy==1.19.5
[INFO] DOCKER> Downloading numpy-1.19.5-cp36-cp36m-manylinux2010_x86_64.whl (14.8 MB)
[INFO] DOCKER> Collecting scipy==1.5.4
[INFO] DOCKER> Downloading scipy-1.5.4-cp36-cp36m-manylinux1_x86_64.whl (25.9 MB)
[INFO] DOCKER> Collecting scikit-learn==0.24.2
[INFO] DOCKER> Downloading scikit_learn-0.24.2-cp36-cp36m-manylinux2010_x86_64.whl (22.2 MB)
[INFO] DOCKER> Collecting dill==0.3.4
[INFO] DOCKER> Downloading dill-0.3.4-py2.py3-none-any.whl (86 kB)
[INFO] DOCKER> Collecting joblib==1.1.1
[INFO] DOCKER> Downloading joblib-1.1.1-py2.py3-none-any.whl (309 kB)
[INFO] DOCKER> Collecting jep==4.2.0
[INFO] DOCKER> Downloading jep-4.2.0.tar.gz (3.0 MB)
[INFO] DOCKER> Preparing metadata (setup.py): started
[INFO] DOCKER> Preparing metadata (setup.py): finished with status 'done'
[INFO] DOCKER> Collecting threadpoolctl>=2.0.0
[INFO] DOCKER> Downloading threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
[INFO] DOCKER> Using legacy 'setup.py install' for jep, since package 'wheel' is not installed.
[INFO] DOCKER> Installing collected packages: numpy, threadpoolctl, scipy, joblib, scikit-learn, jep, dill
[INFO] DOCKER> Running setup.py install for jep: started
[INFO] DOCKER> Running setup.py install for jep: finished with status 'error'
[INFO] DOCKER> [91m    ERROR: Command errored out with exit status 1:
     command: /prereqs/pyenv/versions/3.6.13/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_waf9dr3/jep_1f9fe8d5a48043f2a1c1e7aae859f4b9/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_waf9dr3/jep_1f9fe8d5a48043f2a1c1e7aae859f4b9/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ftu2lizj/install-record.txt --single-version-externally-managed --compile --install-headers /prereqs/pyenv/versions/3.6.13/include/python3.6m/jep
         cwd: /tmp/pip-install-_waf9dr3/jep_1f9fe8d5a48043f2a1c1e7aae859f4b9/
    Complete output (72 lines):
    numpy include found at /prereqs/pyenv/versions/3.6.13/lib/python3.6/site-packages/numpy/core/include
    running install
    running build
    running setup_java
    Using JAVA_HOME: /usr/lib/jvm/java-11-amazon-corretto.x86_64/
    running build_java
    /usr/lib/jvm/java-11-amazon-corretto.x86_64/bin/javac -deprecation -d build/java -h build/include -classpath src src/main/java/jep/JepConfig.java src/main/java/jep/ClassList.java src/main/java/jep/Jep.java src/main/java/jep/Interpreter.java src/main/java/jep/LibraryLocator.java src/main/java/jep/DirectNDArray.java src/main/java/jep/SharedInterpreter.java src/main/java/jep/NamingConventionClassEnquirer.java src/main/java/jep/MainInterpreter.java src/main/java/jep/PyConfig.java src/main/java/jep/PyMethod.java src/main/java/jep/AbstractNDArray.java src/main/java/jep/SubInterpreterOptions.java src/main/java/jep/ClassEnquirer.java src/main/java/jep/SubInterpreter.java src/main/java/jep/NDArray.java src/main/java/jep/JepAccess.java src/main/java/jep/Run.java src/main/java/jep/Proxy.java src/main/java/jep/JepException.java src/main/java/jep/python/PyObject.java src/main/java/jep/python/MemoryManager.java src/main/java/jep/python/InvocationHandler.java src/main/java/jep/python/PyPointer.java src/main/java/jep/python/PyCallable.java src/main/java/jep/python/PyBuiltins.java
    src/main/java/jep/SubInterpreterOptions.java:61: error: unmappable character (0xE2) for encoding US-ASCII
         * allocator state. Otherwise it will use (share) the main interpreter???s.
                                                                              ^
    src/main/java/jep/SubInterpreterOptions.java:61: error: unmappable character (0x80) for encoding US-ASCII
         * allocator state. Otherwise it will use (share) the main interpreter???s.
                                                                               ^
    src/main/java/jep/SubInterpreterOptions.java:61: error: unmappable character (0x99) for encoding US-ASCII
         * allocator state. Otherwise it will use (share) the main interpreter???s.
                                                                                ^
    src/main/java/jep/SubInterpreterOptions.java:114: error: unmappable character (0xE2) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                    ^
    src/main/java/jep/SubInterpreterOptions.java:114: error: unmappable character (0x80) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                     ^
    src/main/java/jep/SubInterpreterOptions.java:114: error: unmappable character (0x99) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                      ^
    src/main/java/jep/SubInterpreterOptions.java:114: error: unmappable character (0xE2) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                                             ^
    src/main/java/jep/SubInterpreterOptions.java:114: error: unmappable character (0x80) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                                              ^
    src/main/java/jep/SubInterpreterOptions.java:114: error: unmappable character (0x99) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                                               ^
    src/main/java/jep/SubInterpreterOptions.java:128: error: unmappable character (0xE2) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                    ^
    src/main/java/jep/SubInterpreterOptions.java:128: error: unmappable character (0x80) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                     ^
    src/main/java/jep/SubInterpreterOptions.java:128: error: unmappable character (0x99) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                      ^
    src/main/java/jep/SubInterpreterOptions.java:128: error: unmappable character (0xE2) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                                             ^
    src/main/java/jep/SubInterpreterOptions.java:128: error: unmappable character (0x80) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                                              ^
    src/main/java/jep/SubInterpreterOptions.java:128: error: unmappable character (0x99) for encoding US-ASCII
         * If this is false then the sub-interpreter???s threading module won???t be
                                                                               ^
    src/main/java/jep/SubInterpreterOptions.java:162: error: unmappable character (0xE2) for encoding US-ASCII
         * this is false use (share) the main interpreter???s GIL. When this is true
                                                         ^
    src/main/java/jep/SubInterpreterOptions.java:162: error: unmappable character (0x80) for encoding US-ASCII
         * this is false use (share) the main interpreter???s GIL. When this is true
                                                          ^
    src/main/java/jep/SubInterpreterOptions.java:162: error: unmappable character (0x99) for encoding US-ASCII
         * this is false use (share) the main interpreter???s GIL. When this is true
                                                           ^
    src/main/java/jep/SubInterpreterOptions.java:163: error: unmappable character (0xE2) for encoding US-ASCII
         * use the sub-interpreter???s own GIL.
                                  ^
    src/main/java/jep/SubInterpreterOptions.java:163: error: unmappable character (0x80) for encoding US-ASCII
         * use the sub-interpreter???s own GIL.
                                   ^
    src/main/java/jep/SubInterpreterOptions.java:163: error: unmappable character (0x99) for encoding US-ASCII
         * use the sub-interpreter???s own GIL.
                                    ^
    21 errors
    error: command '/usr/lib/jvm/java-11-amazon-corretto.x86_64/bin/javac' failed with exit status 1
    ----------------------------------------

[INFO] DOCKER> [91mERROR: Command errored out with exit status 1: /prereqs/pyenv/versions/3.6.13/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_waf9dr3/jep_1f9fe8d5a48043f2a1c1e7aae859f4b9/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_waf9dr3/jep_1f9fe8d5a48043f2a1c1e7aae859f4b9/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ftu2lizj/install-record.txt --single-version-externally-managed --compile --install-headers /prereqs/pyenv/versions/3.6.13/include/python3.6m/jep Check the logs for full command output.

[pvb05]

It looks like non US-ASCII apostrophe used in the SubIntepreterOptions.java javadocs. Can you please suggest options to resolve it?

[bsteffensmeier]

You should be able to resolve this by setting the JAVA_TOOL_OPTIONS env var to include -Dfile.encoding=UTF-8.

I will convert the apostrophe to U+0027 in the next jep release so this won't be an issue in the future.