search

yt-project / yt

Main yt repository
http://yt-project.org
Other
464 stars 276 forks source link

Fresh yt installation not importing #2892

Closed Provider10 closed 3 years ago

Provider10 commented 4 years ago

Bug report

Bug summary

yt fails to import on a fresh development installation. The error is that ModuleNotFoundError: No module named 'yt.utilities.lib.misc_utilities' Similar to Issue 2685

Code for reproduction Installed using pip install git+git://github.com/yt-project/yt.git

import yt

Actual outcome

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-2d2292a375dc> in <module>
----> 1 import yt

~/.conda/envs/ytgit/lib/python3.8/site-packages/yt/__init__.py in <module>
     63 )
     64 
---> 65 from yt.fields.api import (
     66     field_plugins,
     67     DerivedField,

~/.conda/envs/ytgit/lib/python3.8/site-packages/yt/fields/api.py in <module>
      1 # from . import species_fields
----> 2 from . import (
      3     angular_momentum,
      4     astro_fields,
      5     cosmology_fields,

~/.conda/envs/ytgit/lib/python3.8/site-packages/yt/fields/angular_momentum.py in <module>
      1 import numpy as np
      2 
----> 3 from yt.utilities.lib.misc_utilities import (
      4     obtain_position_vector,
      5     obtain_relative_velocity_vector,

ModuleNotFoundError: No module named 'yt.utilities.lib.misc_utilities'

Expected outcome

Version Information

Installed using pip install git+git://github.com/yt-project/yt.git

Thanks!

matthewturk commented 4 years ago

Hi! One reason this often happens is if it isn't built. But, pip installing it like this should build. Can you provide the full log output from the pip install?

On Fri, Sep 4, 2020 at 10:32 AM Raziq Noorali notifications@github.com wrote:

Bug report

Bug summary

yt fails to import on a fresh development installation. The error is that ModuleNotFoundError: No module named 'yt.utilities.lib.misc_utilities'

Code for reproduction Installed using pip install git+git://github.com/yt-project/yt.git

Paste your code here##import yt

Actual outcome

If applicable, paste the console output here

# #

Expected outcome

Version Information

  • Operating System: CentOS Linux 7 (Core)x86_64
  • Python Version: 3.8.5
  • yt version: 4.0.dev0
  • Other Libraries (if applicable): N/A

Installed using pip install git+git://github.com/yt-project/yt.git

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/yt-project/yt/issues/2892, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAVXO7JCM3FCUAY7FJ3CGTSEECCRANCNFSM4QYVEXEQ .

Provider10 commented 4 years ago

Hi @matthewturk. I apologize for the delay. In solidarity with the graduate student strike at the University of Michigan, I have not crossed imposed picket lines and am getting to this now. Please let me know if you need anything else!

log from pip install: 

$ pip install git+git://github.com/yt-project/yt.git
Collecting git+git://github.com/yt-project/yt.git
  Cloning git://github.com/yt-project/yt.git to /tmp/pip-req-build-m9_kw_3f
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting IPython>=1.0
  Downloading ipython-7.18.1-py3-none-any.whl (786 kB)
     |████████████████████████████████| 786 kB 5.6 MB/s 
Collecting unyt>=2.7.2
  Using cached unyt-2.7.2-py2.py3-none-any.whl (97 kB)
Collecting numpy>=1.10.4
  Downloading numpy-1.19.2-cp38-cp38-manylinux2010_x86_64.whl (14.5 MB)
     |████████████████████████████████| 14.5 MB 53.4 MB/s 
Requirement already satisfied: setuptools>=19.6 in ./.conda/envs/ytgit_test/lib/python3.8/site-packages (from yt==4.0.dev0) (49.6.0.post20200814)
Collecting matplotlib>=1.5.3
  Downloading matplotlib-3.3.2-cp38-cp38-manylinux1_x86_64.whl (11.6 MB)
     |████████████████████████████████| 11.6 MB 85.2 MB/s 
Collecting sympy>=1.2
  Using cached sympy-1.6.2-py3-none-any.whl (5.8 MB)
Collecting decorator
  Using cached decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Collecting pexpect>4.3; sys_platform != "win32"
  Using cached pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Collecting pygments
  Downloading Pygments-2.7.1-py3-none-any.whl (944 kB)
     |████████████████████████████████| 944 kB 83.9 MB/s 
Collecting traitlets>=4.2
  Downloading traitlets-5.0.4-py3-none-any.whl (98 kB)
     |████████████████████████████████| 98 kB 5.3 MB/s 
Collecting prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0
  Downloading prompt_toolkit-3.0.7-py3-none-any.whl (355 kB)
     |████████████████████████████████| 355 kB 87.6 MB/s 
Collecting pickleshare
  Using cached pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)
Collecting jedi>=0.10
  Using cached jedi-0.17.2-py2.py3-none-any.whl (1.4 MB)
Collecting backcall
  Using cached backcall-0.2.0-py2.py3-none-any.whl (11 kB)
Collecting python-dateutil>=2.1
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting cycler>=0.10
  Using cached cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Collecting pillow>=6.2.0
  Using cached Pillow-7.2.0-cp38-cp38-manylinux1_x86_64.whl (2.2 MB)
Collecting kiwisolver>=1.0.1
  Using cached kiwisolver-1.2.0-cp38-cp38-manylinux1_x86_64.whl (92 kB)
Requirement already satisfied: certifi>=2020.06.20 in ./.conda/envs/ytgit_test/lib/python3.8/site-packages (from matplotlib>=1.5.3->yt==4.0.dev0) (2020.6.20)
Collecting pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Processing ./.cache/pip/wheels/29/2c/1c/d2e4580cde2743b0aef389e936ac21a2db92921ddbca53faa1/mpmath-1.1.0-py3-none-any.whl
Collecting ptyprocess>=0.5
  Using cached ptyprocess-0.6.0-py2.py3-none-any.whl (39 kB)
Collecting ipython-genutils
  Using cached ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB)
Collecting wcwidth
  Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting parso<0.8.0,>=0.7.0
  Using cached parso-0.7.1-py2.py3-none-any.whl (109 kB)
Collecting six>=1.5
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Building wheels for collected packages: yt
  Building wheel for yt (PEP 517) ... done
  Created wheel for yt: filename=yt-4.0.dev0-py3-none-any.whl size=1857377 sha256=648b2cef24207a5d158b0c61c76a0f8723c97e6a235036c578870ae2e4197267
  Stored in directory: /tmp/pip-ephem-wheel-cache-p6pus6eo/wheels/de/5c/c8/8c939833284c1c193ca0f3dda036b9ea66c6e4089174dace55
Successfully built yt
Installing collected packages: decorator, ptyprocess, pexpect, pygments, ipython-genutils, traitlets, wcwidth, prompt-toolkit, pickleshare, parso, jedi, backcall, IPython, numpy, mpmath, sympy, unyt, six, python-dateutil, cycler, pillow, kiwisolver, pyparsing, matplotlib, yt
Successfully installed IPython-7.18.1 backcall-0.2.0 cycler-0.10.0 decorator-4.4.2 ipython-genutils-0.2.0 jedi-0.17.2 kiwisolver-1.2.0 matplotlib-3.3.2 mpmath-1.1.0 numpy-1.19.2 parso-0.7.1 pexpect-4.8.0 pickleshare-0.7.5 pillow-7.2.0 prompt-toolkit-3.0.7 ptyprocess-0.6.0 pygments-2.7.1 pyparsing-2.4.7 python-dateutil-2.8.1 six-1.15.0 sympy-1.6.2 traitlets-5.0.4 unyt-2.7.2 wcwidth-0.2.5 yt-4.0.dev0
cphyc commented 4 years ago

What is the output of the following two commands?

which pip
which python

Also, it may prove useful to provide the log using pip install --verbose git+git://github.com/yt-project/yt.git.

Provider10 commented 4 years ago

Sure!

(ytgit_test) [rnoorali@gl-login2 ~]$ which pip
~/.conda/envs/ytgit_test/bin/pip
(ytgit_test) [rnoorali@gl-login2 ~]$ which python
~/.conda/envs/ytgit_test/bin/python

The verbose log is quite long, soft the sake of readability I attached it here pip_log.txt

cphyc commented 4 years ago

Sure!

(ytgit_test) [rnoorali@gl-login2 ~]$ which pip
~/.conda/envs/ytgit_test/bin/pip
(ytgit_test) [rnoorali@gl-login2 ~]$ which python
~/.conda/envs/ytgit_test/bin/python

The verbose log is quite long, soft the sake of readability I attached it here pip_log.txt

Alright, this looks definitely like a bug on yt's side: yt relies on some files that need to be compiled (for higher efficiency). However, for some reason they do not get compiled when one installs using the command you tried (I can reproduce this on my side).

In the meantime, you should be able to install with the following commands:

$ cd /path/to/where/yt/will/be
$ git clone https://github.com/yt-project/yt.git
$ cd yt
$ pip install -e .
cphyc commented 4 years ago

So after some debugging, it turns out that yt cannot be installed unless -e is specified in the pip command, i.e.

pip install -e git+https://github.com/yt-project/yt.git#egg=yt   # works
pip install git+https://github.com/yt-project/yt.git#egg=yt      # does not compile the cython files

This stems from the fact that pip install whatever without -e does not call our custom version of build_ext.build_extensions.

cphyc commented 4 years ago

Hi @Provider10 , would it be possible to try out the bugfix I submitted and tell me if this works for you? In order to do so, use the following script

pip install git+git://github.com/cphyc/yt.git@bugfix/2892     #note the different URL!
Provider10 commented 4 years ago

Hi @cphyc. The above command gave the following output:

Collecting git+git://github.com/cphyc/yt.git@bugfix/2892
  Cloning git://github.com/cphyc/yt.git (to revision bugfix/2892) to /tmp/pip-req-build-8pqa2yj1
  WARNING: Did not find branch or tag 'bugfix/2892', assuming revision or ref.
ERROR: Command errored out with exit status 1: git checkout -q bugfix/2892 Check the logs for full command output.

Since I see the above was merged I also checked the current master branch (4d9ab24) and got a new error when I try to import yt:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-2d2292a375dc> in <module>
----> 1 import yt

~/.conda/envs/ytgit_test3/lib/python3.8/site-packages/yt/__init__.py in <module>
     63 )
     64 
---> 65 from yt.fields.api import (
     66     field_plugins,
     67     DerivedField,

~/.conda/envs/ytgit_test3/lib/python3.8/site-packages/yt/fields/api.py in <module>
      1 # from . import species_fields
----> 2 from . import (
      3     angular_momentum,
      4     astro_fields,
      5     cosmology_fields,

~/.conda/envs/ytgit_test3/lib/python3.8/site-packages/yt/fields/angular_momentum.py in <module>
      1 import numpy as np
      2 
----> 3 from yt.utilities.lib.misc_utilities import (
      4     obtain_position_vector,
      5     obtain_relative_velocity_vector,

~/.conda/envs/ytgit_test3/lib/python3.8/site-packages/yt/utilities/lib/misc_utilities.pyx in init yt.utilities.lib.misc_utilities()

~/.conda/envs/ytgit_test3/lib/python3.8/site-packages/yt/geometry/oct_container.pyx in init yt.geometry.oct_container()

~/.conda/envs/ytgit_test3/lib/python3.8/site-packages/yt/geometry/oct_visitors.pyx in init yt.geometry.oct_visitors()

~/.conda/envs/ytgit_test3/lib/python3.8/site-packages/yt/geometry/selection_routines.pyx in init yt.geometry.selection_routines()

ModuleNotFoundError: No module named 'yt.utilities.lib.image_samplers'

I'd like to recommend that this issue be reopened

cphyc commented 4 years ago

That is embarrassing. Can you confirm that if you clone the yt repository and install via pip install -e . it works?

Provider10 commented 4 years ago

Lol sorry about that! When installing as in your above suggestion, I get the following error message (send via text log to save space). pip_log.txt

Thanks for taking the time to go through this with me! I really appreciate it

cphyc commented 4 years ago

The pip command you should try pip install -e . after having cloned the repository. The log you sent me used pip install --verbose git+git://github.com/yt-project/yt.git instead. Note that the latter should also work, so you definitely found a bug!

Provider10 commented 4 years ago

Ah, my apologies for sending you a novel-length log then. I'm afraid I'm not versed enough in yt to be able to attempt a bigfix, but please let me know how I can help!

RevathiJambunathan commented 4 years ago

Hello! I get this error too after a fresh installation of the most recent master branch of yt, and when I try to import yt.

pip version : pip 20.2.3 python version : Python 3.7.9 setuptools version : setuptools 50.3.0

When i git cloned yt, and checkout out master, pip3 install -e . --user

I get the following error

    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/rjambunathan/Documents/SOFTWARES/yt/setup.py'"'"'; __file__='"'"'/home/rjambunathan/Documents/SOFTWARES/yt/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps --user --prefix=
         cwd: /home/rjambunathan/Documents/SOFTWARES/yt/
    Complete output (29 lines):
    Using OpenMP to compile parallel extensions
    running develop
    WARNING: The user site-packages directory is disabled.
    error: can't create or remove files in install directory

    The following error occurred while trying to add or remove files in the
    installation directory:

        [Errno 13] Permission denied: '/usr/local/lib/python3.7/dist-packages/test-easy-install-25156.write-test'

    The installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:

        /usr/local/lib/python3.7/dist-packages/

    Perhaps your account does not have write access to this directory?  If the
    installation directory is a system-owned directory, you may need to sign in
    as the administrator or "root" account.  If you do not have administrative
    access to this machine, you may wish to choose a different installation
    directory, preferably one that is listed in your PYTHONPATH environment
    variable.

    For information on other options, you may wish to consult the
    documentation at:

      https://setuptools.readthedocs.io/en/latest/easy_install.html

    Please make the appropriate changes for your system and try again.

    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/rjambunathan/Documents/SOFTWARES/yt/setup.py'"'"'; __file__='"'"'/home/rjambunathan/Documents/SOFTWARES/yt/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps --user --prefix= Check the logs for full command output.

Please let me know if there is any other detail that may help to fix this or if there is something that I can fix in my python environment setup.

ax3l commented 4 years ago

In the meantime, you should be able to install with the following commands: ... pip install -e .

Just a note as @RevathiJambunathan is encountering the same issue as the OP and we debugged it together offline a bit.

I would focus on the issue with pip install git+git://github.com/yt-project/yt.git because:

this is our life now

python3 -m pip wheel . python3 -m pip install *whl


(apply `--user` as needed if you are outside of virtual envs)
- alternatively, someone suggested `python3 -m pip install --no-use-pep517 -e .` to  get back editable installs [[3]](https://github.com/pypa/pip/issues/7953#issuecomment-676600130) (pep517 is the future, consider this is a hack)
  - note that uninstalling already present editable installs can be tricky: [[4]](https://stackoverflow.com/questions/17346619/how-to-uninstall-editable-packages-with-pip-installed-with-e)
- for debugging the issue, the output of
  - `python3 -m pip wheel -v .` might be useful to post here
RevathiJambunathan commented 4 years ago

pip wheel -v . 2>&1 | tee build.log gives build.log

upon "grepping" for error I get :+1: grepping_error.txt

the gcc version is - gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609

ax3l commented 4 years ago

It looks like the GCC 5.4.0 compiler does not recognize the compiled files as C++11 in those setups: https://github.com/yt-project/yt/blob/88597474c2f49d527941ac8d13296044a2dc3f59/yt/utilities/lib/ewahboolarray/ewah.h#L626-L628

  yt/utilities/lib/ewahboolarray/ewah.h:626:31: error: expected ‘,’ or ‘...’ before ‘&&’ token
  yt/utilities/lib/ewahboolarray/ewah.h:626:38: error: invalid constructor; you probably meant ‘EWAHBoolArray<uword> (const EWAHBoolArray<uword>&)’

This might not occur in CI or on newer systems, because GCC 6+ default to C++14: https://gist.github.com/ax3l/53db9fa8a4f4c21ecc5c4100c0d93c94

Checking the history of that file, it looks like #2575 introduced this with an update of EWAHBoolArray, which defaults to C++11 in newer versions.

The solution would probably be to add -std=c++<N> flags explicitly in yt's build system. It's confusing though, because #2575 adds extra_compile_args=["-std=c++11"]) in setup.py. Does it need to be added to further Extensions that include the same header? Or the problem came in with recent changes to the setup.py compiler flags, e.g. #2610

I can reproduce this on newer Ubuntu's by running:

export CC=$(which gcc-5)
export CXX=$(which g++-5)
python3 -m pip wheel -v .

A related challenge/bug in this issue: this is clearly an error in the build process. A build error of a component/extension should abort the whole installation, instead of silently installing an incomplete version of yt.

ax3l commented 4 years ago

Proposed bug fix for compiler compatibility: #2939

A related challenge/bug in this issue: this is clearly an error in the build process. A build error of a component/extension should abort the whole installation, instead of silently installing an incomplete version of yt.

I have no idea how to fix this aspect.

Xarthisius commented 4 years ago

A related challenge/bug in this issue: this is clearly an error in the build process. A build error of a component/extension should abort the whole installation, instead of silently installing an incomplete version of yt.

I have no idea how to fix this aspect.

Try the following:

diff --git a/setupext.py b/setupext.py
index 6b20f8b..4a505a8 100644
--- a/setupext.py
+++ b/setupext.py
@@ -6,7 +6,7 @@ import shutil
 import subprocess
 import sys
 import tempfile
-from concurrent.futures import ThreadPoolExecutor as Pool
+from concurrent.futures import ThreadPoolExecutor
 from distutils import log
 from distutils.ccompiler import CCompiler, new_compiler
 from distutils.errors import CompileError, LinkError
@@ -354,11 +354,23 @@ def create_build_ext(lib_exts, cythonize_aliases):

             ncpus = get_cpu_count()
             if ncpus > 0:
-                with Pool(ncpus) as pool:
-                    pool.map(self.build_extension, self.extensions)
+                with ThreadPoolExecutor(ncpus) as executor:
+                    results = {
+                        executor.submit(self.build_extension, extension): extension
+                        for extension in self.extensions
+                    }
+                for result in results:
+                    result.result()
             else:
                 super().build_extensions()

+        def build_extension(self, extension):
+            try:
+                super().build_extension(extension)
+            except CompileError as exc:
+                print(f"While building '{extension.name}' following error was raised:\n {exc}")
+                raise
+
     class sdist(_sdist):
         # subclass setuptools source distribution builder to ensure cython
         # generated C files are included in source distribution.

Now the main process should die if any child thread died.

ax3l commented 3 years ago

@Xarthisius revisiting you suggestion, should this patch still go into mainline? :)

Xarthisius commented 3 years ago

@ax3l I believe it's already there (see https://github.com/yt-project/yt/pull/2948)