libpython linux issue finding GLIBCXX_3.4.21

[aaelony-catasys]

Able to run Zeroshot classification example using conda env in python but not in libpython-clj. Curious why isn't finding paths needed that it would find in ipython proper.

"ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found"

(ns aa.zeroshot                                                                                                                                                
    (:require [libpython-clj2.python :as py :refer [py. py.- py.. py* py**]]                                                                                   
              aa.python                                                                                                                                        
              [libpython-clj2.require :refer [require-python import-python]]))

;; INFO: Detecting startup info for Python executable /home/aaelony/.conda/envs/py-nlp                                                                         
;; INFO: Startup info            

{:lib-version "3.9",
 :java-library-path-addendum "/home/aaelony/.conda/envs/py-nlp/lib",
 :exec-prefix "/home/aaelony/.conda/envs/py-nlp",
 :executable "/home/aaelony/.conda/envs/py-nlp/bin/python3",
 :libnames ("/home/aaelony/.conda/envs/py-nlp/lib" "python3.9"),
 :prefix "/home/aaelony/.conda/envs/py-nlp",
 :base-prefix "/home/aaelony/.conda/envs/py-nlp",
 :libname "/home/aaelony/.conda/envs/py-nlp/lib",
 :base-exec-prefix "/home/aaelony/.conda/envs/py-nlp",
 :python-home "/home/aaelony/.conda/envs/py-nlp",
 :version [3 9 2],
 :platform "linux"}

(import-python)

(require-python '[transformers :bind-ns])

(def classifier (py. transformers "pipeline" "zero-shot-classification"))

;; Execution error at libpython-clj2.python.ffi/check-error-throw (ffi.clj:670).                                                                               
;; Traceback (most recent call last):                                                                                                                          
;;   File "/home/aaelony/.conda/envs/py-nlp/lib/python3.9/site-packages/transformers/__init__.py", line 2190, in __getattr__                                   
;;     return super().__getattr__(name)                                                                                                                        
;;   File "/home/aaelony/.conda/envs/py-nlp/lib/python3.9/site-packages/transformers/file_utils.py", line 1494, in __getattr__                                 
;;     module = self._get_module(self._class_to_module[name])                                                                                                  
;;   File "/home/aaelony/.conda/envs/py-nlp/lib/python3.9/site-packages/transformers/__init__.py", line 2184, in _get_module                                   
;;     return importlib.import_module("." + module_name, self.__name__)                                                                                        
;;   File "/home/aaelony/.conda/envs/py-nlp/lib/python3.9/importlib/__init__.py", line 127, in import_module                                                   
;;     return _bootstrap._gcd_import(name[level:], package, level)                                                                                             
;;   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import                                                                                           
;;   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load                                                                                        
;;   File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked                                                                                
;;   File "<frozen importlib._bootstrap>", line 680, in _load_unlocked                                                                                         
;;   File "<frozen importlib._bootstrap_external>", line 790, in exec_module                                                                                   
;;   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed                                                                              
;;   File "/home/aaelony/.conda/envs/py-nlp/lib/python3.9/site-packages/transformers/pipelines/__init__.py", line 25, in <module>                              
;;     from ..models.auto.tokenization_auto import AutoTokenizer                                                                                               
;;   File "/home/aaelony/.conda/envs/py-nlp/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py", line 53, in <module>                   
;;     from ..tapas.tokenization_tapas import TapasTokenizer                                                                                                   
;;   File "/home/aaelony/.conda/envs/py-nlp/lib/python3.9/site-packages/transformers/models/tapas/tokenization_tapas.py", line 47, in <module>                 
;;     import pandas as pd                                                                                                                                     
;;   File "/home/aaelony/.conda/envs/py-nlp/lib/python3.9/site-packages/pandas/__init__.py", line 51, in <module>                                              
;;     from pandas.core.api import (                                                                                                                           
;;   File "/home/aaelony/.conda/envs/py-nlp/lib/python3.9/site-packages/pandas/core/api.py", line 31, in <module>                                              
;;     from pandas.core.groupby import Grouper, NamedAgg                                                                                                       
;;   File "/home/aaelony/.conda/envs/py-nlp/lib/python3.9/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>                                  
;;     from pandas.core.groupby.generic import DataFrameGroupBy, NamedAgg, SeriesGroupBy                                                                       
;;   File "/home/aaelony/.conda/envs/py-nlp/lib/python3.9/site-packages/pandas/core/groupby/generic.py", line 65, in <module>                                  
;;     from pandas.core.frame import DataFrame                                                                                                                 
;;   File "/home/aaelony/.conda/envs/py-nlp/lib/python3.9/site-packages/pandas/core/frame.py", line 119, in <module>                                           
;;     from pandas.core import algorithms, common as com, generic, nanops, ops                                                                                 
;;   File "/home/aaelony/.conda/envs/py-nlp/lib/python3.9/site-packages/pandas/core/generic.py", line 111, in <module>                                         
;;     from pandas.core.window import Expanding, ExponentialMovingWindow, Rolling, Window                                                                      
;;   File "/home/aaelony/.conda/envs/py-nlp/lib/python3.9/site-packages/pandas/core/window/__init__.py", line 1, in <module>                                   
;;     from pandas.core.window.ewm import (  # noqa:F401                                                                                                       
;;   File "/home/aaelony/.conda/envs/py-nlp/lib/python3.9/site-packages/pandas/core/window/ewm.py", line 9, in <module>                                        
;;     import pandas._libs.window.aggregations as window_aggregations                                                                                          
;; ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by                                                                         
/home/aaelony/.conda/envs/py-nlp/lib/python3.9/site-packages/pandas/_libs/window/aggregations.cpython-39-x86_64-linux-gnu.so)                                  
;;                                                                                                                                                             

This is using Linux:

$  cat /proc/version 
Linux version 3.10.0-1160.15.2.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Wed Feb 3 15:06:38 UTC 2021

[cnuernber]

I wonder if perhaps the jvm and the python systems were compiled with different versions of glibc. Can conda install java?

[aaelony-catasys]

I think conda could install java (e.g. https://anaconda.org/bioconda/java-jdk), would that matter? What is weird to me is that the python shell seems to find everything it needs. Could it be that linux needs a special way to determine the paths similar to what windows does in libpython-clj2.python.windows ?

[cnuernber]

Did you set LD_LIBRARY_PATH? When you run the Conda python from the command line a lot of stuff is compiled into the executable -- it doesn't actually load the shared library:

https://github.com/conda/conda/issues/9500#issuecomment-565753807

conda compiles a fat executable with a lot of those things compiled in. Aside from that; the libcXX shared library needs to be in LD_LIBRARY_PATH -- for the java executable to work.

When you compile things on some operating systems you have a chance to set the shared library lookup pathways into the executable itself so analyzing the python executable itself may very well yield some insight as to why it can find libc++ and the java executable cannot.

[aaelony-catasys]

I can try that. What is the preferred way to set LD_LIBRARY_PATH (can it be done from cider/emacs?)

[cnuernber]

That depends on your shell scripting environment. It is an environment variable that has to be set before the jvm executable is started.

[aaelony-catasys]

Thank-you!! You fixed it :)

export LD_LIBRARY_PATH="/home/aaelony/.conda/envs/py-nlp/x86_64-conda_cos6-linux-gnu/:/home/aaelony/.conda/envs/py-nlp/lib/:$LD_LIBRARY_PATH"

Exporting LD_LIBRARY_PATH as above prior to running the CIDER session fixes this issue.
I'll try not to forget this next time, but maybe this could happen behind the scenes or hint message somehow?

thanks again!!

[cnuernber]

That is awesome!!

For future reference, here is a way to set that variable that I think is a bit more robust:

export LD_LIBRARY_PATH="$(python3-config --prefix)/lib"