search

enthought / mayavi

3D visualization of scientific data in Python
http://docs.enthought.com/mayavi/mayavi/
Other
1.32k stars 285 forks source link

Failed to install latest version of mayavi with the self-compiled git master version of vtk. #927

Open hongyi-zhao opened 4 years ago

hongyi-zhao commented 4 years ago

Hi,

My environment is: Ubuntu 20.04 + python 3.8.3 (managed by pyenv) + vtk git master version with python wrapper.

I try to install the latest version of mayavi use the following steps:

# Create a virtualenv for mayavi: 
$ pyenv shell 3.8.3
# The vtk wrapper is installed in the base python enironment, hence use the following option:
$ pyenv virtualenv --system-site-packages mayavi
$ pyenv shell --unset
$ pyenv shell mayavi
$ git clone https://github.com/enthought/mayavi.git mayavi.git
$ cd mayavi.git
$ pip install -U pip
$ grep -v '^vtk' requirements.txt | xargs -n1 pip install
$ pip install PyQt5
$ pip install -e .
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Obtaining file:///home/werner/Public/hpc/tools/enthought/mayavi.git
Requirement already satisfied: apptools in /home/werner/.pyenv/versions/3.8.3/envs/mayavi/lib/python3.8/site-packages (from mayavi==4.7.2.dev0) (4.5.0)
Requirement already satisfied: envisage in /home/werner/.pyenv/versions/3.8.3/envs/mayavi/lib/python3.8/site-packages (from mayavi==4.7.2.dev0) (4.9.2)
Requirement already satisfied: numpy in /home/werner/.pyenv/versions/3.8.3/envs/mayavi/lib/python3.8/site-packages (from mayavi==4.7.2.dev0) (1.18.5)
Requirement already satisfied: pyface>=6.1.1 in /home/werner/.pyenv/versions/3.8.3/envs/mayavi/lib/python3.8/site-packages (from mayavi==4.7.2.dev0) (7.0.0)
Requirement already satisfied: pygments in /home/werner/.pyenv/versions/3.8.3/envs/mayavi/lib/python3.8/site-packages (from mayavi==4.7.2.dev0) (2.6.1)
Requirement already satisfied: traits>=4.6.0 in /home/werner/.pyenv/versions/3.8.3/envs/mayavi/lib/python3.8/site-packages (from mayavi==4.7.2.dev0) (6.1.0)
Requirement already satisfied: traitsui>=6.0.0 in /home/werner/.pyenv/versions/3.8.3/envs/mayavi/lib/python3.8/site-packages (from mayavi==4.7.2.dev0) (7.0.0)
ERROR: Could not find a version that satisfies the requirement vtk (from mayavi==4.7.2.dev0) (from versions: none)
ERROR: No matching distribution found for vtk (from mayavi==4.7.2.dev0)

Any hints for this problem?

Regards, HY

mathstuf commented 4 years ago

Did you build hte VTK wheel? I don't think the pip install is going to work without the VTK wheel.

hongyi-zhao commented 4 years ago

Any more hints on doing this? I just only compiled the python wrapper of VTK using the following options:

        -DVTK_WRAP_PYTHON=ON                              \
        -DVTK_PYTHON_VERSION=3
mathstuf commented 4 years ago

You'll need to make the wheel using python setup.py bdist_wheel in the build tree once the VTK build is done. You can then pip install that wheel and mayavi should install just fine once you have it in your venv.

hongyi-zhao commented 4 years ago

I'm still confused on this problem. In detail, with the following options

        -DVTK_WRAP_PYTHON=ON                              \
        -DVTK_PYTHON_VERSION=3

The python VTK package will be built and installed for the venv and I can run import vtk successfully.

For mayavi, considering that the python VTK package already exists in the venv, why must I use the wheel based method you described above?

mathstuf commented 4 years ago

I read one wrong. That way of installation means that pip doesn't know VTK is already installed. Just remove it from the list of packages needed by mayavi (as far as pip knows at least).

hongyi-zhao commented 4 years ago

As you can see, I've removed the vtk from list invoked by pip install:

$ grep -v '^vtk' requirements.txt | xargs -n1 pip install

mathstuf commented 4 years ago

But the pip install -e . still sees it. It needs to actually be gone from requirements.txt I think.

prabhuramachandran commented 4 years ago

What do you get when you do pip list, do you see vtk listed at all? Instead of using pip install -e . can you try python setup.py install? If neither works, try pip install -e --no-deps .

hongyi-zhao commented 4 years ago

But the pip install -e . still sees it. It needs to actually be gone from requirements.txt I think.

No, I tried by commenting out it with #vtkor deleting this line from requirements.txt, but both methods cannot solve the problem.

hongyi-zhao commented 4 years ago

@prabhuramachandran pip list won't list vtk for my case, instead the vtk package is imported to the venv by the virtualenv's --system-site-packages option and actually exists in the base python environment on which the mayavi venv is created with the following command:

$ pyenv virtualenv --system-site-packages mayavi

This will put the following line in the venv's pyvenv.cfg file:

$ grep 'include-system-site-packages' $VIRTUAL_ENV/pyvenv.cfg
include-system-site-packages = true

The python setup.py install command will fail with similar error message: No local packages or working download links found for vtk

As for your 3rd suggested method, the command should be run as following:

$ pip install -e . --no-deps

Though the installation completed successfully, the problem leaves as it was and not solved at all.

$ mayavi2 
Traceback (most recent call last):
  File "/home/werner/.pyenv/versions/test/bin/mayavi2", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/home/werner/.pyenv/versions/3.8.3/envs/test/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3251, in <module>
    def _initialize_master_working_set():
  File "/home/werner/.pyenv/versions/3.8.3/envs/test/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3234, in _call_aside
    f(*args, **kwargs)
  File "/home/werner/.pyenv/versions/3.8.3/envs/test/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3263, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/werner/.pyenv/versions/3.8.3/envs/test/lib/python3.8/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/home/werner/.pyenv/versions/3.8.3/envs/test/lib/python3.8/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/werner/.pyenv/versions/3.8.3/envs/test/lib/python3.8/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'vtk' distribution was not found and is required by mayavi

Regards, HY

prabhuramachandran commented 4 years ago

How about: python setup.py develop --no-deps?

prabhuramachandran commented 4 years ago

In the worst case comment out the VTK requirement inside mayavi/__init__.py. Should I remove this, i.e. remove the vtk requirement in setup.py?

hongyi-zhao commented 4 years ago

How about: python setup.py develop --no-deps?

This won't solve the problem. After the installation, mayavi2 command will throw out the following error and fails to start: pkg_resources.DistributionNotFound: The 'vtk' distribution was not found and is required by mayavi

hongyi-zhao commented 4 years ago

In the worst case comment out the VTK requirement inside mayavi/__init__.py. Should I remove this, i.e. remove the vtk requirement in setup.py?

Commented out the vtk and the problem still can't be solved.

prabhuramachandran commented 4 years ago

That makes no sense. I suspect something else is going on. The only hard requirement is from the setup file which sets the requirements. The rest are dynamic requirements through the import. Can you please do a pip uninstall mayavi first and then do this? Also what was the last error you got?

prabhuramachandran commented 4 years ago

I suspect many of the issues are because you have an older partially installed version in your local site-packages. python setup.py develop will not install the files in site-packages. So if you comment/remove the 'vtk' line from the mayavi/__init__.py pkg resources should never bother you about that requirement unless old files are lying around. So, always first pip uninstall mayavi before trying the options above. I suspect if you rewind and try pip install -e . after cleaning up, that it should work too (once you comment out the requires).

hongyi-zhao commented 4 years ago

Great. The following steps solve the problem and will use the python vtk package I installed for the base python environment:

$ pip uninstall mayavi
# Commented out the vtk line:
$ grep vtk mayavi/__init__.py
    #'vtk'
$ pip install -e .
# See the snapshot below for my case.
$ mayavi2

It seems the vtk shouldn't be put in the mayavi/__init__.py considering that we already installed it through pypi or manually compilation. Some fixes need to be added for dealing with this problem in mayavi's source code.

image

prabhuramachandran commented 4 years ago

This is really a packaging/pip/setup mess for which I am not aware if there is a clean solution. If I remove the vtk line from the requirements, then someone who just installs pip install mayavi will run into trouble not finding VTK and this is the largest use case.

The problem is that your installation of VTK seems to not "declare itself" to pkg_resources which is why it is complaining all the time even though import vtk works. So that issue perhaps needs to be looked at. Did you follow the advice of @mathstuf and build the wheel and install that? That is what I usually do when installing VTK.

hongyi-zhao commented 4 years ago

  1. Considering that compiling and installation of VTK is a heavy and time-consuming job, I only install the python VTK package into the base python environment. This is the reason that the pip list of the real venv cannot find the vtk package while import vtk works due to virtualenv's option --system-site-packages.

  2. Till now, I haven't tried the wheel based packaging method suggested by @mathstuf . But even with this method, I still need to compile and install VTK for all virtuelenvs which need VTK.

prabhuramachandran commented 4 years ago

So are you saying pkg_resources does not honor system-site-packages? That seems like a bug in pkg_resources (setuptools)? That in itself is a major problem because if this breaks all packages using requires will break badly for packages installed in the base environment.

As regards wheels, you only build the wheel once, and you can reinstall that wheel if you want any number of times. Is your base environment installation using a wheel?

hongyi-zhao commented 4 years ago

So are you saying pkg_resources does not honor system-site-packages? That seems like a bug in pkg_resources (setuptools)? That in itself is a major problem because if this breaks all packages using requires will break badly for packages installed in the base environment.

As regards wheels, you only build the wheel once, and you can reinstall that wheel if you want any number of times. Is your base environment installation using a wheel?

No. To be frank, I've zero experience on using the wheel packaging tool.

hongyi-zhao commented 4 years ago

You'll need to make the wheel using python setup.py bdist_wheel in the build tree once the VTK build is done. You can then pip install that wheel and mayavi should install just fine once you have it in your venv.

I'm very confused on the VTKPythonPackage and VTK Python Wrapper. It's obvious that the above method mentioned by you refers which the former. But for using this method, I should first build and install VTK first. But what's the differences between the above two method for interfacing VTK with python?

Any hints for my question?

Regards, HY

prabhuramachandran commented 4 years ago

With the latest VTK, just follow what @mathstuf has said above. https://github.com/enthought/mayavi/issues/927#issuecomment-640605145

hongyi-zhao commented 4 years ago

Is so, the VTK Python Wrapper should not be used at all, so I should set the following: -DVTK_WRAP_PYTHON=OFF

Am I right?

prabhuramachandran commented 4 years ago

No, if you do that, it will not build the VTK wrapper at all! You are using the VTK sources, so all you need to do is run the build and follow the instructions that @mathstuf sent above, i.e. run the python setup.py bdist_wheel to build the wheel. Then install the wheel.

hongyi-zhao commented 4 years ago

Now, I follow the instructions by @mathstuf but when starting mayavi2, I meet the following errors:


$ mayavi2 
Exception occurred in traits notification handler for object: <mayavi.plugins.mayavi_ui_plugin.MayaviUIPlugin object at 0x7f9ad1731450>, trait: application, old value: None, new value: <mayavi.plugins.mayavi_workbench_application.MayaviWorkbenchApplication object at 0x7f9ad17319f0>
Traceback (most recent call last):
  File "/home/werner/.pyenv/versions/3.8.3/envs/vtk/lib/python3.8/site-packages/traits/trait_notifiers.py", line 580, in _dispatch_change_event
    self.dispatch(handler, *args)
  File "/home/werner/.pyenv/versions/3.8.3/envs/vtk/lib/python3.8/site-packages/traits/trait_notifiers.py", line 484, in dispatch
    handler(*args)
  File "/home/werner/.pyenv/versions/3.8.3/envs/vtk/lib/python3.8/site-packages/traits/traits_listener.py", line 412, in handle_simple
    self.next.register(new)
  File "/home/werner/.pyenv/versions/3.8.3/envs/vtk/lib/python3.8/site-packages/traits/traits_listener.py", line 386, in register
    value = getattr(self, type)(new, name, False)
  File "/home/werner/.pyenv/versions/3.8.3/envs/vtk/lib/python3.8/site-packages/traits/traits_listener.py", line 607, in _register_simple
    return next.register(getattr(object, name))
  File "/home/werner/.pyenv/versions/3.8.3/envs/vtk/lib/python3.8/site-packages/envisage/ui/workbench/workbench_application.py", line 140, in _gui_default
    return GUI(splash_screen=self.splash_screen)
  File "/home/werner/Public/hpc/tools/enthought/mayavi.git/mayavi/plugins/mayavi_workbench_application.py", line 118, in _splash_screen_default
    splash_screen = SplashScreen(
  File "/home/werner/.pyenv/versions/3.8.3/envs/vtk/lib/python3.8/site-packages/pyface/base_toolkit.py", line 184, in __init__
    raise NotImplementedError(msg % (toolkit, package, name))
NotImplementedError: the null pyface.ui.null backend doesn't implement splash_screen:SplashScreen
Exception occurred in traits notification handler for object: <mayavi.scripts.mayavi2.MayaviApp object at 0x7f9b85bba360>, trait: application, old value: None, new value: <mayavi.plugins.mayavi_workbench_application.MayaviWorkbenchApplication object at 0x7f9ad17319f0>
Traceback (most recent call last):
  File "/home/werner/.pyenv/versions/3.8.3/envs/vtk/lib/python3.8/site-packages/traits/trait_notifiers.py", line 580, in _dispatch_change_event
    self.dispatch(handler, *args)
  File "/home/werner/.pyenv/versions/3.8.3/envs/vtk/lib/python3.8/site-packages/traits/trait_notifiers.py", line 484, in dispatch
    handler(*args)
  File "/home/werner/.pyenv/versions/3.8.3/envs/vtk/lib/python3.8/site-packages/traits/traits_listener.py", line 412, in handle_simple
    self.next.register(new)
  File "/home/werner/.pyenv/versions/3.8.3/envs/vtk/lib/python3.8/site-packages/traits/traits_listener.py", line 386, in register
    value = getattr(self, type)(new, name, False)
  File "/home/werner/.pyenv/versions/3.8.3/envs/vtk/lib/python3.8/site-packages/traits/traits_listener.py", line 607, in _register_simple
    return next.register(getattr(object, name))
  File "/home/werner/.pyenv/versions/3.8.3/envs/vtk/lib/python3.8/site-packages/envisage/ui/workbench/workbench_application.py", line 140, in _gui_default
    return GUI(splash_screen=self.splash_screen)
  File "/home/werner/Public/hpc/tools/enthought/mayavi.git/mayavi/plugins/mayavi_workbench_application.py", line 118, in _splash_screen_default
    splash_screen = SplashScreen(
  File "/home/werner/.pyenv/versions/3.8.3/envs/vtk/lib/python3.8/site-packages/pyface/base_toolkit.py", line 184, in __init__
    raise NotImplementedError(msg % (toolkit, package, name))
NotImplementedError: the null pyface.ui.null backend doesn't implement splash_screen:SplashScreen
Traceback (most recent call last):
  File "/home/werner/.pyenv/versions/vtk/bin/mayavi2", line 11, in <module>
    load_entry_point('mayavi', 'gui_scripts', 'mayavi2')()
  File "/home/werner/Public/hpc/tools/enthought/mayavi.git/mayavi/scripts/mayavi2.py", line 602, in main
    mayavi.main(sys.argv[1:])
  File "/home/werner/Public/hpc/tools/enthought/mayavi.git/mayavi/plugins/app.py", line 197, in main
    app.run()
  File "/home/werner/Public/hpc/tools/enthought/mayavi.git/mayavi/plugins/mayavi_workbench_application.py", line 73, in run
    gui = self.gui
  File "/home/werner/.pyenv/versions/3.8.3/envs/vtk/lib/python3.8/site-packages/envisage/ui/workbench/workbench_application.py", line 140, in _gui_default
    return GUI(splash_screen=self.splash_screen)
  File "/home/werner/Public/hpc/tools/enthought/mayavi.git/mayavi/plugins/mayavi_workbench_application.py", line 118, in _splash_screen_default
    splash_screen = SplashScreen(
  File "/home/werner/.pyenv/versions/3.8.3/envs/vtk/lib/python3.8/site-packages/pyface/base_toolkit.py", line 184, in __init__
    raise NotImplementedError(msg % (toolkit, package, name))
NotImplementedError: the null pyface.ui.null backend doesn't implement splash_screen:SplashScreen
prabhuramachandran commented 4 years ago

This suggests that you have not installed a suitable UI backend. Install PySide2 or PyQt5 and it should work.

hongyi-zhao commented 4 years ago

I installed both of the above two packages in the venv, but still I meet the same error:

$ pip list | egrep 'PyQt|PySide'
PyQt5                         5.15.0
PyQt5-sip                     12.8.0
PySide2                       5.15.0