Closed arengel closed 1 year ago
I am able to reproduce this. Seems very bad! What the heck?!
I doubt it's related, but I get the same behavior (albeit faster!) if I export USE_PYGEOS=0
before firing up the python prompt.
I found a suggestion on stackoverflow to use faulthandler
to track down where the issue is occurring.
From the full results which are below, it looks like the problem is happening in h3
which is a dependency of timezonefinder
. And indeed, when I import either of those, I get the same segmentation fault.
Confirmed in issue uber/h3-py#313.
import faulthandler
faulthandler.enable()
import timezonefinder
Fatal Python error: Segmentation fault
Current thread 0x0000000100aac580 (most recent call first):
File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 1176 in create_module
File "<frozen importlib._bootstrap>", line 571 in module_from_spec
File "<frozen importlib._bootstrap>", line 674 in _load_unlocked
File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
File "/Users/aengel/mambaforge/envs/test/lib/python3.10/site-packages/h3/_cy/__init__.py", line 16 in <module>
File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 883 in exec_module
File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1078 in _handle_fromlist
File "/Users/aengel/mambaforge/envs/test/lib/python3.10/site-packages/h3/api/basic_int/_binding.py", line 16 in <module>
File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 883 in exec_module
File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
File "/Users/aengel/mambaforge/envs/test/lib/python3.10/site-packages/h3/api/basic_int/_public_api.py", line 9 in <module>
File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 883 in exec_module
File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
File "/Users/aengel/mambaforge/envs/test/lib/python3.10/site-packages/h3/api/basic_int/__init__.py", line 1 in <module>
File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 883 in exec_module
File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1078 in _handle_fromlist
File "/Users/aengel/mambaforge/envs/test/lib/python3.10/site-packages/h3/api/__init__.py", line 3 in <module>
File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 883 in exec_module
File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 992 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
File "/Users/aengel/mambaforge/envs/test/lib/python3.10/site-packages/h3/__init__.py", line 3 in <module>
File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 883 in exec_module
File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 992 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
File "/Users/aengel/mambaforge/envs/test/lib/python3.10/site-packages/timezonefinder/timezonefinder.py", line 10 in <module>
File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 883 in exec_module
File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
File "/Users/aengel/mambaforge/envs/test/lib/python3.10/site-packages/timezonefinder/__init__.py", line 2 in <module>
File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 883 in exec_module
File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
File "<stdin>", line 1 in <module>
Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator (total: 13)
zsh: segmentation fault python
Thank you for that sleuthing! It's disturbing that the issue has been open for a month with no resolution.
@arengel - FYI, for internal planning purposes I'm going to edit this issue to include a 'scope' and 'next steps' section. (Thanks for bringing this up!)
It looks like there was some work on the H3 conda-forge feedstock to try and fix this but it petered out.
We'd like to work on this but not necessarily blow up our whole sprint working on this.
Scope
Out of scope:
Next steps
I'm interested in helping but am on WSL, not mac...
Replicated on a new M2 Macbook air, will play around with this a bit...
Seems like it's probably a bunch of dependency futzing. We've considered using conda-lock to lock exact versions of all of our dependencies so this kind of drift after the fact can't happen, if that's something you feel like exploring!
Locking dependancies makes total sense and is why I've always used a pinned requirements.txt
or Pipfile.lock
. I'll give this a shot!
We have some annoying to build dependencies that need C/C++/Fortran extensions (e.g. geospatial stuff) and some non-python dependencies that conda
can satisfy without needing to resort to using Docker or other heavier weight environment management systems (e.g. the snappy compression library, SQLite binaries) and it's also nice to use the conda supplied Python rather than hope that the user has the right version of Python available on their system. At least in development we've primarily used conda
for environment virtualization, with the PUDL package getting installed within that environment by pip
.
(also I have a PR (#2479) open right now to move all our project metadata / build stuff into pyproject.toml
and get rid of setup.py
finally.
I ran into some problems with conda-lock not playing nicely with our git-pinned pip requirements. But I think we may have gotten rid of some and/or conda-lock may have gotten better.
Zane Selvans @.***>于2023年4月3日 周一下午6:01写道:
(also I have a PR (#2479 https://github.com/catalyst-cooperative/pudl/pull/2479) open right now to move all our project metadata / build stuff into pyproject.toml and get rid of setup.py finally.
— Reply to this email directly, view it on GitHub https://github.com/catalyst-cooperative/pudl/issues/2426#issuecomment-1495043252, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATBKMU3A67W53FWM5WPQTTW7NCDNANCNFSM6AAAAAAWBW4HQM . You are receiving this because you were assigned.Message ID: @.***>
All right, I got something working!
It looks like the conda-forge maintainers will fix the root cause but until then this workaround uses pip to install a more recent version of h3-py
.
I also implemented conda-lock which I think will be a great way to better track dependancies. However I mamba is so much faster that might make sense to let users decide if they prefer more deterministic locking or faster, sloppier locking
# mamba does not support creating from an environment.yml file, so create and then update:
mamba env create --name test
mamba env update -n test --file environment.yml
conda-lock -f environment.yml -p osx-64 -p linux-64 -p osx-arm64
(base) ➜ pudl git:(main) ✗ conda-lock install --name fromlock conda-lock.yml
WARNING:conda_lock.conda_lock:WARNING: installation of pip dependencies is only supported by the 'conda-lock install' command. Other tools may silently ignore them. For portability, we recommend using the newer unified lockfile format (i.e. removing the --kind=explicit argument.
INFO:root:Downloading and Extracting Packages
INFO:root:Downloading and Extracting Packages
INFO:root:Preparing transaction: ...working... done
INFO:root:Verifying transaction: ...working... done
INFO:root:Executing transaction: ...working... done
INFO:root:Collecting h3@ https://files.pythonhosted.org/packages/51/4b/31ed29f5dca2093f6e29707dc21b0cf7071de4623ade7491bd2ee191617d/h3-3.7.6-cp310-cp310-macosx_11_0_arm64.whl#sha256=8bf1e080b9a47774754834e7f10155f3d2e3542bf895488a0519b2ae7d5b15db
INFO:root: Using cached h3-3.7.6-cp310-cp310-macosx_11_0_arm64.whl (904 kB)
INFO:root:Installing collected packages: h3
INFO:root: Attempting uninstall: h3
INFO:root: Found existing installation: h3 3.7.4
INFO:root: Uninstalling h3-3.7.4:
INFO:root: Successfully uninstalled h3-3.7.4
INFO:root:Successfully installed h3-3.7.6
INFO:root:
(base) ➜ pudl git:(main) ✗ conda activate fromlock
(fromlock) ➜ pudl git:(main) ✗ python
Python 3.10.10 | packaged by conda-forge | (main, Mar 24 2023, 20:12:31) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pudl
/Users/nelsonauner/mambaforge/envs/fromlock/lib/python3.10/site-packages/pudl/analysis/spatial.py:7: UserWarning: Shapely 2.0 is installed, but because PyGEOS is also installed, GeoPandas will still use PyGEOS by default for now. To force to use and test Shapely 2.0, you have to set the environment variable USE_PYGEOS=0. You can do this before starting the Python process, or in your code before importing geopandas:
import os
os.environ['USE_PYGEOS'] = '0'
import geopandas
In a future release, GeoPandas will switch to using Shapely by default. If you are using PyGEOS directly (calling PyGEOS functions on geometries from GeoPandas), this will then stop working and you are encouraged to migrate from PyGEOS to Shapely 2.0 (https://shapely.readthedocs.io/en/latest/migration_pygeos.html).
import geopandas as gpd
>>> #works! no segfault
This should hopefully be fixed by v2022.11.30.post1 which I will get up on conda-forge
some time tomorrow (once their bot has picked up the new PyPI release)
I may have spoken too soon here. Unfortunately the only version of h3-py
that is available via conda
is 3.7.4, which is the one that was giving us problems.
$ mamba search h3-py
# Name Version Build Channel
h3-py 3.7.4 py310h0f1eb42_1 conda-forge
h3-py 3.7.4 py311ha397e9f_1 conda-forge
h3-py 3.7.4 py38h2b1e499_1 conda-forge
h3-py 3.7.4 py39h23fbdae_1 conda-forge
Maybe h3-py 3.7.4 only works on Python 3.10? My next guess would be to try to reproduce this issue in 3.10 vs. 3.11...
https://github.com/uber/h3-py/blob/master/CHANGELOG.md#374---2022-04-14
For those following along, this is still an issue given that h3-py is still not updated on conda (but is on pip).
This issue can be "patched" simply by, after running the conda install, using pip to update h3-py:
# Install and activate environment
➜ mamba create --name test python=3.10 catalystcoop.pudl
➜ conda activate test
# Load python and attempt to import `pudl`:
(test) ➜ python
Python 3.10.10 | packaged by conda-forge | (main, Mar 24 2023, 20:12:31) [Clang 14.0.6 ] on darwin
>>> import pudl
....
[1] 96656 segmentation fault python
# Show that we're using virtual env pip and update the h3 module:
(test) ➜ pudl git:(pandas-2.0) ✗ which pip
/Users/nelsonauner/mambaforge/envs/test/bin/pip
(test) ➜ pudl git:(pandas-2.0) ✗ pip install "h3>=3.7.6,<3.8"
Collecting h3<3.8,>=3.7.6
Using cached h3-3.7.6-cp310-cp310-macosx_11_0_arm64.whl (904 kB)
Installing collected packages: h3
Attempting uninstall: h3
Found existing installation: h3 3.7.4
Uninstalling h3-3.7.4:
Successfully uninstalled h3-3.7.4
Successfully installed h3-3.7.6
# Now it works
@zaneselvans @arengel
Great news - https://anaconda.org/conda-forge/h3-py was updated over the weekend to 3.7.6 and now works totally fine:
Zane - can you close this if everything here looks fine? I would at some point love to pick up convo on versioning we had in https://github.com/catalyst-cooperative/pudl/pull/2497 but lower priority than upgrading pandas 2.0
(base) ➜ pudl git:(dev) ✗ mamba create --name test-update python=3.10 catalystcoop.pudl
# ... + h3-py 3.7.6 py310h0f1eb42_0 conda-forge/osx-arm64 298kB
(base) ➜ pudl git:(dev) ✗ conda activate test-update
(test-update) ➜ pudl git:(dev) ✗ python
Python 3.10.10 | packaged by conda-forge | (main, Mar 24 2023, 20:12:31) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pudl
/Users/nelsonauner/mambaforge/envs/test-update/lib/python3.10/site-packages/pudl/analysis/spatial.py:7: UserWarning: Shapely 2.0 is installed, but because PyGEOS is also installed, GeoPandas will still use PyGEOS by default for now. To force to use and test Shapely 2.0, you have to set the environment variable USE_PYGEOS=0. You can do this before starting the Python process, or in your code before importing geopandas:
import os
os.environ['USE_PYGEOS'] = '0'
import geopandas
In a future release, GeoPandas will switch to using Shapely by default. If you are using PyGEOS directly (calling PyGEOS functions on geometries from GeoPandas), this will then stop working and you are encouraged to migrate from PyGEOS to Shapely 2.0 (https://shapely.readthedocs.io/en/latest/migration_pygeos.html).
import geopandas as gpd
>>> print("it works, hooray!!")
it works, hooray!!
Oh great!
Does this mean that the post1
release we tried to get going on conda is no longer necessary, since it'll automatically pick up the newer version of h3-py
?
@zaneselvans Yes, no longer necessary!
Okay great. Sounds like this is fixed. Hopefully we don't have to deal with any more of this software dependency stuff going forward with the bigger data distributions!
Describe the bug
I just re-created a new conda environment where I installed PUDL from conda-forge. When I import PUDL in that environment, it results in a segmentation fault.
The last time I installed PUDL from conda-forge, this didn't happen, though I can't say for sure when that would have been.
Bug Severity
How badly is this bug affecting you?
To Reproduce
Create the new environment, install PUDL, activate the environment, and launch a python console.
In the console, import PUDL
The console then produces this output and exits.
Software Environment?
mamba create --name test python=3.10 catalystcoop.pudl
Full set of packages and versions:
Additional context
Add any other context about the problem here.
======== edit from @jdangerx