search

grumpyhome / grumpy

Grumpy is a Python to Go source code transcompiler and runtime.
Apache License 2.0
420 stars 18 forks source link

Install grumpy-tools before building grumpy-runtime #140

Closed abitrolly closed 4 years ago

abitrolly commented 5 years ago

pip2 install grumpy-runtime fails with the ERROR https://github.com/grumpyhome/grumpy/issues/113#issuecomment-521527773 because it relies on grumpy-tools.

grumpy-tools packages is specified in setup.py as a dependency, and it doesn't work, because for the build process to succeed, the dependency should already exist at the time setup.py is executed.

The official workaround is pyproject.toml from PEP-518 https://github.com/pypa/pip/issues/2381

abitrolly commented 5 years ago

This still doesn't fix the error. pip claims that it installs build dependencies.

$ pipenv clean
$ pipenv run python setup.py sdist
...
copying pyproject.toml -> grumpy-runtime-0.3.0
...
$  pipenv run pip install dist/grumpy-runtime-0.3.0.tar.gz 
Processing ./dist/grumpy-runtime-0.3.0.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting grumpy-tools>=0.3.0 (from grumpy-runtime==0.3.0)
...

But then it still fails with error.

...
Building wheels for collected packages: grumpy-runtime
  Building wheel for grumpy-runtime (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/bin/python /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpRJ3HDO                                                                                                                   
       cwd: /tmp/pip-req-build-DJvU2j                                                                                                                                   
  Complete output (757 lines):                                                                                                                                          
  running bdist_wheel                                                                                                                                                   
  running build                                                                                                                                                         
...
      from grumpy_tools import cli                                                                                                                                      
  ImportError: No module named grumpy_tools                                                                                                                             
  Traceback (most recent call last):                                                                                                                                    
    File "/tmp/pip-req-build-DJvU2j/build/bin/pydeps", line 5, in <module>                                                                                              
      from grumpy_tools import cli                                                                                                                                      
  ImportError: No module named grumpy_tools                                                                                                                             
  Re-executing[2]: make run --debug=bjm -r                                                                                                                              
  GNU Make 4.2.1                                                                                                                                                        
  Built for x86_64-redhat-linux-gnu                                            
...
      Successfully remade target file 'build/src/grumpy/type.go'.                                                                                                       
    Must remake target 'build/pkg/linux_amd64/grumpy.a'.                                                                                                                
    Successfully remade target file 'build/pkg/linux_amd64/grumpy.a'.                                                                                                   
     File 'build/pkg/linux_amd64/__python__/os.a' does not exist.                                                                                                       
       File 'build/src/__python__/os/module.go' does not exist.                                                                                                         
      Must remake target 'build/src/__python__/os/module.go'.                                                                                                           
  Traceback (most recent call last):                                                                                                                                    
    File "/tmp/pip-req-build-DJvU2j/build/bin/grumpc", line 5, in <module>                                                                                              
      from grumpy_tools import cli                                                                                                                                      
  ImportError: No module named grumpy_tools                                                                                                                             
  make: *** [build/stdlib.mk:2181: build/src/__python__/os/module.go] Error 1                                                                                           
  Traceback (most recent call last):                                                                                                                                    
    File "/home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>                  
      main()                                                                                                                                                            
...
    File "/usr/lib64/python2.7/subprocess.py", line 190, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['echo "print \'Make Runtime Success\'" | make run --debug=bjm -r']' returned non-zero exit status 2
  ----------------------------------------
  ERROR: Failed building wheel for grumpy-runtime
  Running setup.py clean for grumpy-runtime
Failed to build grumpy-runtime
ERROR: Could not build wheels for grumpy-runtime which use PEP 517 and cannot be installed directly
alanjds commented 5 years ago

Thanks for your contribution.

The tests are not passing with Go=stable. Would you mind do also fix this before the merge?

abitrolly commented 5 years ago

Looks like Python build environment that is set for setup.py is not passed by subprocess to make. Or perhaps make strips the environment. Could also be the problem with shebangs https://github.com/pypa/setuptools/issues/847

abitrolly commented 5 years ago

sys.path in setup.py

  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib/python2.7/site-packages/pip/_vendor/pep517
  /tmp/pip-build-env-OE7ua4/site
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib/python27.zip
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/plat-linux2
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/lib-tk
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/lib-old
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/lib-dynload
  /usr/lib64/python2.7
  /usr/lib/python2.7
  /tmp/pip-build-env-OE7ua4/overlay/lib/python2.7/site-packages
  /tmp/pip-build-env-OE7ua4/overlay/lib64/python2.7/site-packages
  /tmp/pip-build-env-OE7ua4/normal/lib/python2.7/site-packages
  /tmp/pip-build-env-OE7ua4/normal/lib64/python2.7/site-packages

sys.path in subprocess.check_call with shell=True.

  /tmp/pip-req-build-nK1nBr/build/bin
  /tmp/pip-req-build-nK1nBr/build/lib/python2.7/site-packages
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib/python27.zip
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/plat-linux2
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/lib-tk
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/lib-old
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/lib-dynload
  /usr/lib64/python2.7
  /usr/lib/python2.7
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib/python2.7/site-packages

/tmp/pip-build-env-OE7ua4 with installed grumpy-tools from build-system requires is gone, and /tmp/pip-req-build-nK1nBr without them is present.

abitrolly commented 5 years ago

Reported the issue upstream in setuptools. Because build environment is not a proper virtualenv or venv, and setup.py is executed inprocess, the sys.path is not passed to subprocess and it picks up parent pip environment.

The workaround for now is to serialize and pass sys.path to subprocess explicitly as PYTHONPATH environment variable. It will probably mix with virtualenv from pip and if/when upstream fixes that, the workaround should be removed.

abitrolly commented 5 years ago

Passing sys.path as PYTHONPATH fails.

  Reading makefiles...                                                             
  Traceback (most recent call last):                                               
    File "/home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/site.py", line 791, in <module>                                                      
      main()                                                                       
    File "/home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/site.py", line 781, in main                                                          
      execsitecustomize()                                                          
    File "/home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/site.py", line 553, in execsitecustomize                                             
      import sitecustomize                                                         
    File "/tmp/pip-build-env-FWZ6LX/site/sitecustomize.py", line 22, in <module>   
      assert not path in sys.path
abitrolly commented 5 years ago

Trying to forcefully inject just a single build path /tmp/pip-build-env-tqb_4I/normal/lib64/python2.7/site-packages doesn't work. It passed as PYTHONPATH to make, but not available when pydeps is run.

Maybe parent vritualenv is killing it.

abitrolly commented 5 years ago

@alanjds ready for review. Install works ok.

$ git clone -b fix-install git@github.com:abitrolly/grumpy.git
$ cd grumpy/grumpy-runtime-src
$ pipenv --python 2.7
$ pipenv run pip install .
alanjds commented 4 years ago

Thanks @abitrolly. I bet this was a pain to tackle.

abitrolly commented 4 years ago

Took quite a bit a time, indeed.