justineyster commented 3 years ago

We have a tool called detect-secrets that depends on ibm-db. Many of our users can't properly install/import it right now, and are experiencing the error below (including myself). We've had to pull the ibm-db dependency from our installation and make it an optional feature, but we'd prefer to see this fixed permanently at the source. It's not the first time we've seen a similar bug. Thank you!

OS (and OS of all users seeing the bug, so far): Mac OS

Traceback (most recent call last):
  File "/Users/justineyster/.pyenv/versions/3.8.6/bin/detect-secrets", line 33, in <module>
    sys.exit(load_entry_point('detect-secrets==0.13.1+ibm.29.dss', 'console_scripts', 'detect-secrets')())
  File "/Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/site-packages/detect_secrets/main.py", line 40, in main
    plugins = initialize.from_parser_builder(
  File "/Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/site-packages/detect_secrets/plugins/common/initialize.py", line 38, in from_parser_builder
    from_plugin_classname(
  File "/Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/site-packages/detect_secrets/plugins/common/initialize.py", line 181, in from_plugin_classname
    klass = import_plugins(plugin_filenames)[plugin_classname]
  File "/Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/site-packages/detect_secrets/plugins/common/util.py", line 46, in import_plugins
    module = import_module('detect_secrets.plugins.{}'.format(module_name))
  File "/Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/site-packages/detect_secrets/plugins/db2.py", line 17, in <module>
    raise ie
  File "/Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/site-packages/detect_secrets/plugins/db2.py", line 7, in <module>
    import ibm_db
ImportError: dlopen(/Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/site-packages/ibm_db.cpython-38-darwin.so, 2): Symbol not found: ___cxa_throw_bad_array_new_length
  Referenced from: /Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/site-packages/clidriver/lib/libdb2.dylib
  Expected in: /usr/lib/libstdc++.6.dylib
 in /Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/site-packages/clidriver/lib/libdb2.dylib
Justins-MBP:whitewater-detect-secrets justineyster$ detect-secrets scan --update .secrets.baseline --db2-scan
Traceback (most recent call last):
  File "/Users/justineyster/.pyenv/versions/3.8.6/bin/detect-secrets", line 33, in <module>
    sys.exit(load_entry_point('detect-secrets==0.13.1+ibm.29.dss', 'console_scripts', 'detect-secrets')())
  File "/Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/site-packages/detect_secrets/main.py", line 40, in main
    plugins = initialize.from_parser_builder(
  File "/Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/site-packages/detect_secrets/plugins/common/initialize.py", line 38, in from_parser_builder
    from_plugin_classname(
  File "/Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/site-packages/detect_secrets/plugins/common/initialize.py", line 181, in from_plugin_classname
    klass = import_plugins(plugin_filenames)[plugin_classname]
  File "/Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/site-packages/detect_secrets/plugins/common/util.py", line 46, in import_plugins
    module = import_module('detect_secrets.plugins.{}'.format(module_name))
  File "/Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/site-packages/detect_secrets/plugins/db2.py", line 17, in <module>
    raise ie
  File "/Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/site-packages/detect_secrets/plugins/db2.py", line 7, in <module>
    import ibm_db
ImportError: dlopen(/Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/site-packages/ibm_db.cpython-38-darwin.so, 2): Symbol not found: ___cxa_throw_bad_array_new_length
  Referenced from: /Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/site-packages/clidriver/lib/libdb2.dylib
  Expected in: /usr/lib/libstdc++.6.dylib
 in /Users/justineyster/.pyenv/versions/3.8.6/lib/python3.8/site-packages/clidriver/lib/libdb2.dylib

justineyster commented 3 years ago

I tried using it with multiple versions of Python using pyenv and saw similar results.

imavo commented 3 years ago

https://github.com/ibmdb/python-ibmdb/issues/540

amukherjee28 commented 3 years ago

We had a recent upgrade of the clidriver and look the old issue is haunting us again. I will check on the at the earliest and come back with an update.

Thanks.

amukherjee28 commented 3 years ago

Hi,

This issue is caused due to the version mismatch of the libstdc++.6.dylib library.

The clidriver downloaded along with the ibm_db driver has a dependency on this particular library. So we are shipping this library along the clidriver for mac in order to resolve the dependency. However for few users this library dependency in met from a different path /usr/lib.

Now this causes the dependency to break and hence the error.

So my suggestion is to try out the following options.

rename the library in /usr/lib so that clidriver picks up the dependency from the shipped library along with clidriver.
setting DYLD_LIBRARY_PATH environmental variable to clidriver/lib folder to allow the driver to pick up the library dependency form the mentioned PATH.

Please let me know if this could be worked out.

Thanks.

drewmullen commented 3 years ago

$ sudo mv /usr/lib/libstdc++.6.dylib{,.bak}
mv: rename /usr/lib/libstdc++.6.dylib to /usr/lib/libstdc++.6.dylib.bak: Operation not permitted

it made me extremely nervous to move this lib and then my mac denied me anyways so...

im also running into this error. any recommendations asap would be great

drewmullen commented 3 years ago

for a quick workaround, this problem does not appear in the python docker containers: docker run -it --name pythondev python:3.8 /bin/bash

rredburn commented 3 years ago

I helped a user fix this by exporting DYLD_LIBRARY_PATH as suggested, thanks.

in .zshrc/.bashrc/etc. (modify for your path):
export DYLD_LIBRARY_PATH=/usr/local/lib/python3.9/site-packages/clidriver/lib:$DYLD_LIBRARY_PATH

gugarrido commented 3 years ago

I fixed that by replacing the files inside clidriver folder with older files a team mate sent me. Just pasting the files in the following path: /lib/python3.8/site-packages/clidriver

justineyster commented 3 years ago

We're looking for something a little more complete than just a workaround, ideally. We support hundreds if not thousands of users of our tool and wouldn't want to handhold everyone through this. Thanks!

(Again though, we've removed ibm_db from our dependencies for the time being and have made the feature that includes it optional. So it's not too pressing. But my dev environment is still broken at the moment.)

amukherjee28 commented 3 years ago

Hi @justineyster

I understand the issue you are facing and this is completely because of library version dependency.

The clidriver that ibm_db is currently using has a dependency on libstdc++.6.dylib library. We are shipping the required version of the library along with the clidriver as well to allow the dependency to work.

However in case the same library also exists in /usr/lib then there is a dependency break.

In order to avoid this scenario that best case scenario is setting the DYLD_LIBRARY_PATH to the clidriver/lib path.

From the issue that you are facing, looks like we have to handle the same while ibm_db is getting installed. I will have a check on this and come back in case that works.

Thanks.

craigcurtin commented 3 years ago

Hi @amukherjee28, I have same issue when trying to install the ibm_db package. I'm on Python 3.8 Please advise how to work around this lib issue to install the ibm_db package for the short term, assuming there is a longer term fix. I've had some colleagues auggest "$ brew install db2" first ... is this correct? How does one install python3-devl on mac-os ? Thanks, Craig

jaykodeveloper commented 3 years ago

I had a similar issue when I was trying to install the ibm_db. I had to spend many days digging into it but no luck and finally reached to here. I hope this issue get solved as soon as possible!

IgorCherepanov commented 3 years ago

I had a similar issue when I was trying to install the ibm_db. I had to spend many days digging into it but no luck and finally reached to here. I hope this issue get solved as soon as possible!

+1. I cannot resolve the issue as well.

amukherjee28 commented 3 years ago

Looks like we have check the library being shipped and check on the packaging again.

I will work on this and have an update at the earliest.

Thanks.

bimalkjha commented 3 years ago

@rredburn @drewmullen @justineyster @craigcurtin @jaykodeveloper This error comes when system has an old version of libstdc++.6.dylib which comes with older version of gcc compiler. If you upgrade the installed gcc to latest version, the latest libstdc++.6.dylib will get installed and then ibm_db will not throw any error. Recommended version of gcc is >= 4.8.3

Alternatively, setting environment variable using export DYLD_LIBRARY_PATH=/usr/local/lib/python3.9/site-packages/clidriver/lib:$DYLD_LIBRARY_PATH command will force to use the libstdc++ from clidriver instead of the one from old version of gcc. Thanks.

drewmullen commented 3 years ago

@bimalkjha all due respect, i have an up to date OS... im not going to update gcc because im guessing thats going to have repercussions. the pathing does not work for me when using pyenv (which is think is the incompatibility here):

$ export DYLD_LIBRARY_PATH=/Users/drmullen/.pyenv/versions/db2/lib/python3.8/site-packages/clidriver/lib/:$DYLD_LIBRARY_PATH
$ python
...
>>> import ibm_db
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/Users/drmullen/.pyenv/versions/db2/lib/python3.8/site-packages/ibm_db.cpython-38-darwin.so, 2): Symbol not found: ___cxa_throw_bad_array_new_length
  Referenced from: /Users/drmullen/.pyenv/versions/db2/lib/python3.8/site-packages/clidriver/lib/libdb2.dylib
  Expected in: /usr/lib/libstdc++.6.dylib
 in /Users/drmullen/.pyenv/versions/db2/lib/python3.8/site-packages/clidriver/lib/libdb2.dylib

i did notice on an ubuntu VM that i have, the install fails when using a venv but works when using system python3. best practice is to never use system python but for the time being thats what im resorting to on ubuntu

for mac users, i did manage to get this to work using

$ pyenv deactivate
$ python3 -m venv venv
$ source venv/bin/activate
$ python --version
Python 3.9.2
$ which pip
/Users/drmullen/venv/bin/pip
$ pip install ibm_db
$ python
>>> import ibm_db

^ without having to export DYLD path

amukherjee28 commented 3 years ago

Hi,

After considering every options possible we have updated the clidriver from our end to include the new library where the loading path of libstdc++.so.6 has been updated.

@rredburn @drewmullen @justineyster @craigcurtin @jaykodeveloper can you please uninstall the ibm_db driver and install again to allow the new clidriver to be downloaded and test your applications.

This time you should not get the error as the library is now loaded with new dependency path and has been corrected.

Please let me know your observations on the same.

Thanks.

drewmullen commented 3 years ago

$ pyenv virtualenv 3.8.2 db2test # new env
$ pyenv activate db2test
$ python -m ibm_db
/Users/drmullen/.pyenv/versions/db2test/bin/python: No module named ibm_db
$ pip install ibm_db==3.0.3
$ python
>>> import ibm_db
>>>

hallelujah. tyvm

craigcurtin commented 3 years ago

hmm, still have issue ... pip install --no-cache-dir ibm_db==3.0.3 Collecting ibm_db==3.0.3 Downloading ibm_db-3.0.3.tar.gz (794 kB) |████████████████████████████████| 794 kB 1.6 MB/s Using legacy 'setup.py install' for ibm-db, since package 'wheel' is not installed. Installing collected packages: ibm-db Running setup.py install for ibm-db ... error ERROR: Command errored out with exit status 1: command: /Users/dt230133/GitLab/pyFWA/venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/k6/mr1x_4xn60v2nylyqmj_ghh48tl07n/T/pip-install-hf6w2y8o/ibm-db_ac48c9813d6e45c1a18be4d16b7a563d/setup.py'"'"'; file='"'"'/private/var/folders/k6/mr1x_4xn60v2nylyqmj_ghh48tl07n/T/pip-install-hf6w2y8o/ibm-db_ac48c9813d6e45c1a18be4d16b7a563d/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/k6/mr1x_4xn60v2nylyqmj_ghh48tl07n/T/pip-record-jzhxfk5l/install-record.txt --single-version-externally-managed --compile --install-headers /Users/dt230133/GitLab/pyFWA/venv/include/site/python3.8/ibm-db cwd: /private/var/folders/k6/mr1x_4xn60v2nylyqmj_ghh48tl07n/T/pip-install-hf6w2y8o/ibm-db_ac48c9813d6e45c1a18be4d16b7a563d/ Complete output (458 lines): Detected 64-bit Python

amukherjee28 commented 3 years ago

@craigcurtin I hope you are in mac platform. Can you install using only pip install ibm_db command and ignore the --no-cache-dir option.

Thanks.

oscarsan commented 3 years ago

This fix works! I was having same problem pip install ibm-db==3.0.3 --no-cache-dir

illeatmyhat commented 3 years ago

pip uninstall -y ibm-db
pip install ibm-db --no-cache-dir

This also worked for me.

craigcurtin commented 3 years ago

hi @amukherjee28, I've been fighting with this ... it appears to be an issue with the extentions needed for the package. I've upgraded the python-dev-tools, setuptools and a bunch of other things ... I've also tried downgrading the ibm_db to the previous versions with out any success. Any other insights appreciated! I have the latest DB2 driver installed on my MacOS 11.2.3 ... I'm kind of out of ideas. Craig building 'ibm_db' extension creating build/temp.macosx-10.14.6-x86_64-3.8 clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -Iclidriver/include -I/Users/dt230133/GitLab/pyFWA/venv/include -I/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c ibm_db.c -o build/temp.macosx-10.14.6-x86_64-3.8/ibm_db.o In file included from ibm_db.c:27: In file included from /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:11: In file included from /Library/Developer/CommandLineTools/usr/lib/clang/12.0.0/include/limits.h:21: In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/limits.h:63: /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/cdefs.h:807:2: error: Unsupported architecture

error Unsupported architecture

 ^
In file included from ibm_db.c:27:
In file included from /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:11:
In file included from /Library/Developer/CommandLineTools/usr/lib/clang/12.0.0/include/limits.h:21:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/limits.h:64:
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/machine/limits.h:8:2: error: architecture not supported
#error architecture not supported
 ^
In file included from ibm_db.c:27:

bimalkjha commented 3 years ago

@craigcurtin You are getting compilation error during install while original problem for this issue was post install error by "import ibm_db" command. So, would suggest to open a new issue with details.

Your error is:

/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/cdefs.h:807:2: error: Unsupported architecture
#error Unsupported architecture
^

It hints your hardware is incompatible. Check on other system. Also, open file MacOSX10.15.sdk/usr/include/sys/cdefs.h and check the architecture of hardware. Good to post issue in macos forum. Thanks.

ibmdb / python-ibmdb

ibm-db broken for many users #608

error Unsupported architecture