search

ibmdb / python-ibmdb

Automatically exported from code.google.com/p/ibm-db
Apache License 2.0
307 stars 192 forks source link

ibm_db crashing on M1 MacBook Pro when running inside VS Code #864

Closed jamsden closed 1 year ago

jamsden commented 1 year ago

Steps to Reproduce:

  1. Set the environment variable DYLD_LIBRARY_PATH in both ~/.zprofile and in a .env fill in the root of my project folder.
  2. start VS Code with code . from that folder
  3. run any simple ibm_db test program in a .py file in vscode, note that it succeeds, indicating that ibm_db is installed and configured properly
  4. run a similar ibm_db test within a jupyter notebook in vscode results in a kernel crash: Cannot execute code, session has been disposed. Please try restarting the Kernel. The Kernel crashed while executing code in the the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click here for more info. View Jupyter log for further deta
  5. run the following: 
    import os
print(os.environ.get('DYLD_LIBRARY_PATH'))

    The result is None.

Likely cause: VS Code is not running in a hardened runtime, or does not have entitlements com.apple.security.cs.disable-library-validation and com.apple.security.cs.allow-dyld-environment-variables set to true. So SIP is causing all DYLD_* environment variables to be stripped. Even if you set that environment variable in the notebook, SIP prevents it from being propagated to spawned processes. This is likely causing ibm_db to crash because of the missing libraries.

bimalkjha commented 1 year ago

@jamsden What is the error that you are getting? Please paste the full error here. Also, share the location of gcc lib in your system. Do you have /usr/local/lib/gcc/12? Or share the directory path that contains libstdc++.6.dylib. Thanks.

bimalkjha commented 1 year ago

@jamsden Your vscode might be of arm64 version and not Intel Only version. ibm_db works with intel-only version of vscode. Check output of : file `which code`command. If you see issue with Intel-Only version of vscode too, then please share the screen shot of the error along with terminal so that we can see how you reproduce the error. In the above shared output, it is too big and lots of strike lines which make is difficult to read and understand which command caused the problem. The errors you pasted is during installation time or run time of ibm_db? It seems you are trying to install something but what is getting installed is not clear. Thanks.

bimalkjha commented 1 year ago
  1. ibm_db works properly in a .py file executed in VS Code
  2. ibm_db works properly in a jupyter notebook run from the jupyter notebook command line. DYLD_LIBRARY_PATH is defined in the jupyther notebook
  3. ibm_db only crashes when run in a jupyter notebook in VS Code. In this case, DYLD_LIBRARY_PATH is not defined in the jupyter notebook. It appears to be being stripped out by M1 security protections.

Since, point 1 and 2 is working fine, so there is no issue with ibm_db installation and using it. For point 3, google and check how to make DYLD_LIBRARY_PATH effective. I can share some commands to get rid of DYLD_LIBRARY_PATH. For that we need output of below commands from terminal:

pip3 --version => note down the location of site_packages
cd site_packages
otool -L ibm_db.*.so
cd clidriver/lib
otool -L libdb2.dylib

Thanks.

jamsden commented 1 year ago

Clarification in item 3: ibm_db only crashes in a Jupyter notebook run in VS code that is using a virtual environment. If the notebook kernel is set to an applicable python version (i.e., 3.9.13 using x86_64 architecture) it works fine. If you create a virtual environment with that same python version, ibm_db crashes because DYLD_LIBRARY_PATH is being striped out of the defined environment variables (i.e., in a .env file) in this case. 

% source venv/bin/activate
(venv) (base) % python --version
Python 3.9.13

site-packages % otool -L ibm_db.*.so
ibm_db.cpython-39-darwin.so:
    @loader_path/clidriver/lib/libdb2.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

 otool -L libdb2.dylib
libdb2.dylib:
    libdb2.dylib (compatibility version 0.0.0, current version 0.0.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1677.104.0)
    /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
    /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/local/Homebrew/Cellar/gcc@12/12.3.0/lib/gcc/12/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.25.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
    /usr/local/lib/gcc/8/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

f.y.i:
cat .env
 cat .env
DYLD_LIBRARY_PATH=/usr/local/lib/gcc/12:/Users/jamsden/Library/CloudStorage/Box-Box/Users/jamsden/Developer/analytics/venv/lib/python3.9/site-packages/clidriver/lib
LD_LIBRARY_PATH=/usr/local/lib/gcc/12:/Users/jamsden/Library/CloudStorage/Box-Box/Users/jamsden/Developer/analytics/venv/lib/python3.9/site-packages/clidriver/lib

In the Jupyter notebook, LD_LIBRARY_PATH is defined in the environment, but DYLD_LD_LIBRARY path isn't its being stripped out possibly because of entitlements.

Hope this helps.

bimalkjha commented 1 year ago

@jamsden How you installed jupyter in vscode? By looking on the crash log, ~/.vscode/extensions/ms-toolsai.jupyter-2023.5.1101742258-darwin-arm64 seems causing this issue. Instead of ms-toolsai.jupyter-2023.5.1101742258-darwin-arm64, it should be something like ms-toolsai.jupyter-2023.5.1101742258-darwin-amd64 or ms-toolsai.jupyter-2023.5.1101742258-darwin-x86_64. This jupyter notebook is of arm64 version. You should install x86_64 version(Intel Only) of jupyter notebook and try. The output of otool -L commands looks good. LD_LIBRARY_PATH does not get read on MacOS and gets ignored. To remove dependency on DYLD_LIBRARY_PATH, you can execute below commands:

cd ..../site_packages/clidriver/lib
intall_name_tool -change /usr/local/lib/gcc/8/libgcc_s.1.dylib /usr/local/Homebrew/Cellar/gcc@12/12.3.0/lib/gcc/12/libgcc_s.1.dylib libdb2.dylib

Thanks.

jamsden commented 1 year ago

I used the vs code marketplace to add python and Jupyter notebook support. The Jupyter I used was installed as part of the virtual environment. I assumed vs code could run in arm64 while executing a kernel that is x86_64, that seems to be the case.

LD_LIBRARY_PATH was just an example environment variable I used to make sure .env was being read properly. I didn't expect it to have any impact on ibm_db. That's how I found DYLD_LIBRARY_PATH is being stripped out vs. not read at all.

I'll try your suggestion to remove the dependency on DYLD_LIBRARY_PATH wit install_name_tool as soon as I can, that looks like a good possibility.

bimalkjha commented 1 year ago

@jamsden Let us know if install_name_tool has solved your issue? Thanks.