Segmentation fault upon importing gdxpds with conda environment

mattirish commented 4 years ago

I use conda for package management and install things with pip only when necessary, such as gdxpds. Despite creating new conda environments with Python versions ranging from 3.4 to 3.6, reinstalling gdxpds with each environment's pip, and then installing the appropriate GAMS bindings with version 26.1, I a segmentation fault every time I try to import gdxpds.

I fear it may be hard to diagnose, but I'm currently unable to use gdxpds, and had this issue on one of several virtual environments I had installed before recently reimaging my machine to upgrade its OS.

Here's the output of pytest --pyargs gdxpds:

mirish$ pytest --pyargs gdxpds
======================================================================================================================================= test session starts =======================================================================================================================================
platform darwin -- Python 3.6.10, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /Users/mirish/projects2/rpm
collecting ... Fatal Python error: Segmentation fault

Current thread 0x00000001092c8dc0 (most recent call first):
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/gdxcc/gdxcc.py", line 268 in gdxFree
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/gdxpds/__init__.py", line 70 in load_gdxcc
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/gdxpds/__init__.py", line 74 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 665 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 941 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 941 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/py/_path/local.py", line 701 in pyimport
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/_pytest/python.py", line 513 in _importtestmodule
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/_pytest/python.py", line 445 in _getobj
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/_pytest/python.py", line 263 in obj
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/_pytest/python.py", line 461 in _inject_setup_module_fixture
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/_pytest/python.py", line 448 in collect
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/_pytest/runner.py", line 264 in <lambda>
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/_pytest/runner.py", line 244 in from_call
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/_pytest/runner.py", line 264 in pytest_make_collect_report
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/_pytest/runner.py", line 382 in collect_one_node
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/_pytest/main.py", line 681 in genitems
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/_pytest/main.py", line 684 in genitems
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/_pytest/main.py", line 490 in _perform_collect
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/_pytest/main.py", line 452 in perform_collect
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/_pytest/main.py", line 257 in pytest_collection
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/_pytest/main.py", line 246 in _main
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/_pytest/main.py", line 191 in wrap_session
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/lib/python3.6/site-packages/_pytest/config/__init__.py", line 125 in main
  File "/Users/mirish/opt/anaconda3/envs/gdxpds_test/bin/pytest", line 11 in <module>
Segmentation fault: 11

It does seem to be gdxcc-related, since the segfault occurs in gdxcc.py.

Will report back if I figure anything out.

jebob commented 4 years ago

Importing gdxcc works for me, importing gdxpds doesn't. Nice find!

jahanbani commented 4 years ago

I have the same issue. Any solutions? importing gdxcc works for me, when I import gdxpds, I get segmentation fault.

jahanbani commented 3 years ago

@mattirish
Hi Matt. Any idea how to solve this issue? I have tried anything I could think of to no avail. Thanks

bpalmintier commented 1 year ago

What is the latest on this issue. I am also having this problem now

bpalmintier commented 1 year ago

Note from some partial looking down this rabbit hole...

It seems the issue is with not being able to find GAMS. But I have GAMS installed (v 34.3) and it is in my path.
Even if it wasn't this should not cause a segfault. At the very least some kind of python try...catch should be possible so folks can have some clue what happened.
I also have so far traced the issue down to a call in __init__.py: from gdxpds.tools import Error. Specifically the following steps are the minimum I have found to reproduce:
```
$ conda create -n gdxpds_bug python=3
$ conda activate gdxpds_bug
$ pip install gdxpds
$ python
>>> from gdxpds.tools import Error
```
Result: Segmentation fault: 11

Tested with gdxpds v1.3, conda 22.9.0 which installed python 3.11.3, under MacOS

bpalmintier commented 1 year ago

Note: with GAMS now fully uninstalled, I no longer get a segfault (partial win?) To fully uninstall on MacOS you have to run sudo bash /Library/Frameworks/GAMS.framework/Versions/{MAJOR VERSION NUMBER TO REMOVE}/Resources/uninstall.sh for all of the different versions on the system.

bpalmintier commented 1 year ago

Further update. It seems this may have to do with some combination (might be one or the other) of multiple GAMS versions being installed and not fully uninstalling an old one and/or mixed up python/conda environment. I was finally able to get things to work again by completely deleting and then later reinstalling GAMS and creating a clean, only pip installed conda environment (made with conda for python, but then pip for all other packages)

mattirish commented 1 year ago

Fancy seeing you here, Bryan! Thanks for picking up this old, dusty torch and relighting it. I'm still using GAMS at Our Next Energy so this may come in handy!

Leaving this open in case somebody comes along and wants to actually fix the bug.

vcelloho commented 9 months ago

I encountered this variant of this issue on ARM based Macs. The issue in that instance was caused when a user had version of Anaconda Python and GAMS that were compiled using different instruction sets. The resolution was to ensure that an ARM based version of GAMS was available. If you aren't licensed for a version of GAMS that includes an ARM based installer a successful work around was to install both, set the path so that the older licensed version was used for optimization, and that for gdxpds the newer one was added using the GAMS_DIR environmental variable.

NREL / gdx-pandas

Segmentation fault upon importing gdxpds with conda environment #73