search

oracle / graalpython

GraalPy – A high-performance embeddable Python 3 runtime for Java
https://www.graalvm.org/python/
Other
1.24k stars 108 forks source link

KeyError: PEP517_BUILD_BACKEND #231

Closed oroppas closed 3 years ago

oroppas commented 3 years ago

graalpython can install packages of wheel. However, packages of sdist fails.

Cython

graalpython -m pip install --user Cython
Collecting Cython
  WARNING: Cache entry deserialization failed, entry ignored
  Downloading Cython-0.29.24-py2.py3-none-any.whl (979 kB)
     |████████████████████████████████| 979 kB 1.3 MB/s 
Installing collected packages: Cython
  WARNING: The scripts cygdb, cython and cythonize are installed in '/u/cd4mcq/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed Cython-0.29.24

numpy

graalpython -m pip install --user numpy
Collecting numpy
  Downloading numpy-1.21.1.zip (10.3 MB)
     |████████████████████████████████| 10.3 MB 32 kB/s 
Looking for Graal Python patches for numpy
We have patches to make this package work on GraalVM for some version(s).
If installing or running fails, consider using one of the versions that we have patches for:
    numpy-1.16.4.patch
  Installing build dependencies ... done
  WARNING: Missing build requirements in pyproject.toml for numpy from https://files.pythonhosted.org/packages/0b/a7/e724c8df240687b5fd62d8c71f1a6709d455c4c09432c7412e3e64f4cbe5/numpy-1.21.1.zip#sha256=dff4af63638afcc57a3dfb9e4b26d434a7a602d225b42d746ea7fe2edf1342fd.
  WARNING: The project does not specify a build backend, and pip cannot fall back to setuptools without 'setuptools>=40.8.0' and 'wheel'.
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /home/ryuta/local/graal/languages/python/bin/graalpython /home/ryuta/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmp7suvzs2a
       cwd: /tmp/pip-install-3_rmbwgq/numpy_1202fdecf92e4aa5a1389b0ec12b3b4e
  Complete output (12 lines):
  Traceback (most recent call last):
    File "/home/ryuta/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 280, in <module '_in_process.py'>
      main()
    File "/home/ryuta/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/ryuta/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 108, in get_requires_for_build_wheel
      backend = _build_backend()
    File "/home/ryuta/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 83, in _build_backend
      ep = os.environ['PEP517_BUILD_BACKEND']
    File "/home/ryuta/packages/graalvm/graal/graal/sdk/mxbuild/linux-amd64/GRAALVM_8052F32AFF_JAVA11/graalvm-8052f32aff-java11-21.3.0-dev/languages/python/lib-python/3/os.py", line 675, in __getitem__
      raise KeyError(key) from None
  KeyError: PEP517_BUILD_BACKEND
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/0b/a7/e724c8df240687b5fd62d8c71f1a6709d455c4c09432c7412e3e64f4cbe5/numpy-1.21.1.zip#sha256=dff4af63638afcc57a3dfb9e4b26d434a7a602d225b42d746ea7fe2edf1342fd (from https://pypi.org/simple/numpy/) (requires-python:>=3.7). Command errored out with exit status 1: /home/ryuta/local/graal/languages/python/bin/graalpython /home/ryuta/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmp7suvzs2a Check the logs for full command output.
  WARNING: Cache entry deserialization failed, entry ignored

pyrsistent

graalpython -m pip install --user pyrsistent
Collecting pyrsistent
  WARNING: Cache entry deserialization failed, entry ignored
  Downloading pyrsistent-0.18.0.tar.gz (104 kB)
     |████████████████████████████████| 104 kB 266 kB/s 
Looking for Graal Python patches for pyrsistent
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /home/ryuta/local/graal/languages/python/bin/graalpython /home/ryuta/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmp9aw0v20p
       cwd: /tmp/pip-install-01vjuzpo/pyrsistent_b613332aa75f4f7da517cbfe13ab7fca
  Complete output (12 lines):
  Traceback (most recent call last):
    File "/home/ryuta/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 280, in <module '_in_process.py'>
      main()
    File "/home/ryuta/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/ryuta/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 108, in get_requires_for_build_wheel
      backend = _build_backend()
    File "/home/ryuta/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 83, in _build_backend
      ep = os.environ['PEP517_BUILD_BACKEND']
    File "/home/ryuta/packages/graalvm/graal/graal/sdk/mxbuild/linux-amd64/GRAALVM_8052F32AFF_JAVA11/graalvm-8052f32aff-java11-21.3.0-dev/languages/python/lib-python/3/os.py", line 675, in __getitem__
      raise KeyError(key) from None
  KeyError: PEP517_BUILD_BACKEND
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/f4/d7/0fa558c4fb00f15aabc6d42d365fcca7a15fcc1091cd0f5784a14f390b7f/pyrsistent-0.18.0.tar.gz#sha256=773c781216f8c2900b42a7b638d5b517bb134ae1acbebe4d1e8f1f41ea60eb4b (from https://pypi.org/simple/pyrsistent/) (requires-python:>=3.6). Command errored out with exit status 1: /home/ryuta/local/graal/languages/python/bin/graalpython /home/ryuta/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmp9aw0v20p Check the logs for full command output.
msimacek commented 3 years ago

@oroppas package installation outside of virtualenv is not yet supported well. Can you reproduce the problem in a virtualenv? I could install pyrsistent fine (I had to do pip install wheel first, but I believe that's expected in a virtualenv created by venv).

oroppas commented 3 years ago

@msimacek I still have the problem. What is your pip version? Mine is

graalpython -m pip --version
pip 21.1.3 from /home/ryuta/.local/lib/python3.8/site-packages/pip (python 3.8)
(venv)
msimacek commented 3 years ago

Mine is: pip 20.1.1 from /home/msimacek/git/graalpython/venv/lib/python3.8/site-packages/pip (python 3.8). That's the one we ship that got copied into the virtualenv. Yours seems to be from you user-site, not from the virtualenv. How did you create the virtualenv? Did you set PYTHONPATH?

oroppas commented 3 years ago

My bad. I forgot to set PYTHONPATH to lib/python3.8/site-packages in venv. Now

graalpython -m pip --version
pip 20.1.1 from /home/ryuta/venv/lib/python3.8/site-packages/pip (python 3.8)
(venv)

Yes, graalpython -m pip install pyrsisten works

graalpython -m pip install pyrsistent
Requirement already satisfied: pyrsistent in ./packages/graalvm/graal/graal/sdk/mxbuild/linux-amd64/GRAALVM_8052F32AFF_JAVA11/graalvm-8052f32aff-java11-21.3.0-dev/languages/python/lib/python3.8/site-packages (0.17.3)
WARNING: You are using pip version 20.1.1; however, version 21.1.3 is available.
You should consider upgrading via the '/home/ryuta/local/graal/languages/python/bin/graalpython -m pip install --upgrade pip' command.
(venv)

It seems that we get KeyError: PEP517_BUILD_BACKEND if we use --user option:

graalpython -m pip install --user pyrsistent
Collecting pyrsistent
  WARNING: Cache entry deserialization failed, entry ignored
  Downloading pyrsistent-0.18.0.tar.gz (104 kB)
     |████████████████████████████████| 104 kB 1.4 MB/s 
Looking for Graal Python patches for pyrsistent
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /home/ryuta/local/graal/languages/python/bin/graalpython /home/ryuta/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpwid1_yl2
       cwd: /tmp/pip-install-e5r45nej/pyrsistent
  Complete output (12 lines):
  Traceback (most recent call last):
    File "/home/ryuta/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module '_in_process.py'>
      main()
    File "/home/ryuta/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/ryuta/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 108, in get_requires_for_build_wheel
      backend = _build_backend()
    File "/home/ryuta/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 83, in _build_backend
      ep = os.environ['PEP517_BUILD_BACKEND']
    File "/home/ryuta/packages/graalvm/graal/graal/sdk/mxbuild/linux-amd64/GRAALVM_8052F32AFF_JAVA11/graalvm-8052f32aff-java11-21.3.0-dev/languages/python/lib-python/3/os.py", line 675, in __getitem__
      raise KeyError(key) from None
  KeyError: PEP517_BUILD_BACKEND
  ----------------------------------------
ERROR: Command errored out with exit status 1: /home/ryuta/local/graal/languages/python/bin/graalpython /home/ryuta/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpwid1_yl2 Check the logs for full command output.
msimacek commented 3 years ago

You shouldn't use --user in a virtualenv, not even in CPython. I'm surprised it let you do that. For me, it says: ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

oroppas commented 3 years ago

Ah. I aliased graalpython to the original installation and every time I type graalpython it did not pick up graalpython under venv/bin. Now

./venv/bin/graalpython -m pip --version
pip 20.1.1 from /home/ryuta/venv/lib/python3.8/site-packages/pip (python 3.8)
(venv)

and I see the expected behavior of pip

./venv/bin/graalpython -m pip install --user pyrsistent
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.
WARNING: You are using pip version 20.1.1; however, version 21.1.3 is available.
You should consider upgrading via the '/home/ryuta/venv/bin/graalpython -m pip install --upgrade pip' command.
(venv)

However, I'm still perplexed by the same error

./venv/bin/graalpython -m pip install pyrsistent
Collecting pyrsistent
  WARNING: Cache entry deserialization failed, entry ignored
  Downloading pyrsistent-0.18.0.tar.gz (104 kB)
     |████████████████████████████████| 104 kB 2.1 MB/s 
Looking for Graal Python patches for pyrsistent
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /home/ryuta/venv/bin/graalpython /home/ryuta/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpudhtwlln
       cwd: /tmp/pip-install-8gfsfqqf/pyrsistent
  Complete output (12 lines):
  Traceback (most recent call last):
    File "/home/ryuta/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module '_in_process.py'>
      main()
    File "/home/ryuta/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/ryuta/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 108, in get_requires_for_build_wheel
      backend = _build_backend()
    File "/home/ryuta/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 83, in _build_backend
      ep = os.environ['PEP517_BUILD_BACKEND']
    File "/home/ryuta/packages/graalvm/graal/graal/sdk/mxbuild/linux-amd64/GRAALVM_8052F32AFF_JAVA11/graalvm-8052f32aff-java11-21.3.0-dev/languages/python/lib-python/3/os.py", line 675, in __getitem__
      raise KeyError(key) from None
  KeyError: PEP517_BUILD_BACKEND
  ----------------------------------------
ERROR: Command errored out with exit status 1: /home/ryuta/venv/bin/graalpython /home/ryuta/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpudhtwlln Check the logs for full command output.
WARNING: You are using pip version 20.1.1; however, version 21.1.3 is available.
You should consider upgrading via the '/home/ryuta/venv/bin/graalpython -m pip install --upgrade pip' command.
Jul 20, 2021 9:46:25 AM org.graalvm.shadowed.org.jline.utils.Log logr
WARNING: Task failed
java.io.IOException: Cannot run program "stty": error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
    at org.graalvm.shadowed.org.jline.utils.ExecHelper.exec(ExecHelper.java:35)
    at org.graalvm.shadowed.org.jline.terminal.impl.ExecPty.doGetConfig(ExecPty.java:175)
    at org.graalvm.shadowed.org.jline.terminal.impl.ExecPty.getAttr(ExecPty.java:87)
    at org.graalvm.shadowed.org.jline.terminal.impl.ExecPty.doSetAttr(ExecPty.java:93)
    at org.graalvm.shadowed.org.jline.terminal.impl.AbstractPty.setAttr(AbstractPty.java:29)
    at org.graalvm.shadowed.org.jline.terminal.impl.AbstractPosixTerminal.doClose(AbstractPosixTerminal.java:76)
    at org.graalvm.shadowed.org.jline.terminal.impl.PosixSysTerminal.doClose(PosixSysTerminal.java:95)
    at org.graalvm.shadowed.org.jline.terminal.impl.AbstractTerminal.close(AbstractTerminal.java:95)
    at org.graalvm.shadowed.org.jline.utils.ShutdownHooks.runTasks(ShutdownHooks.java:55)
    at org.graalvm.shadowed.org.jline.utils.ShutdownHooks.access$000(ShutdownHooks.java:21)
    at org.graalvm.shadowed.org.jline.utils.ShutdownHooks$1.run(ShutdownHooks.java:36)
    at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:567)
    at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: java.io.IOException: error=2, No such file or directory
    at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_io_IOException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0)
    at java.lang.ProcessImpl.forkAndExec(ProcessImpl.java)
    at java.lang.ProcessImpl.<init>(ProcessImpl.java:340)
    at java.lang.ProcessImpl.start(ProcessImpl.java:271)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
    ... 14 more

(venv)
oroppas commented 3 years ago

@msimacek Could you suggest me the best way to debug this error? Thank you.

msimacek commented 3 years ago

I tried it using SVM (I was using JVM before) and now I can reproduce it too, it seems like some SVM-specific problem. I'll try to debug it

msimacek commented 3 years ago

I found the cause - a recent change in running module initialization caused os.environ to be always empty under SVM.

msimacek commented 3 years ago

Fixed by bec9b84c6cf7c0af4b0c2049118843c789133536