search

k2-fsa / k2

FSA/FST algorithms, differentiable, with PyTorch compatibility.
https://k2-fsa.github.io/k2
Apache License 2.0
1.11k stars 213 forks source link

ImportError: /lib64/libm.so.6: version `GLIBC_2.27' not found #1016

Open Aurora-6 opened 2 years ago

Aurora-6 commented 2 years ago

I installed k2 using Conda on centos, but also encounter the import error. Tried the method described in Note , but it didn't work either. What can I do to solve it?

Traceback (most recent call last): File "/root/miniconda3/envs/k2/lib/python3.8/runpy.py", line 185, in _run_module_as_main mod_name, mod_spec, code = _get_module_details(mod_name, _Error) File "/root/miniconda3/envs/k2/lib/python3.8/runpy.py", line 111, in _get_module_details import(pkg_name) File "/root/miniconda3/envs/k2/lib/python3.8/site-packages/k2/init.py", line 28, in raise ImportError( ImportError: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /root/miniconda3/envs/k2/lib/python3.8/site-packages/_k2.cpython-38-x86_64-linux-gnu.so) Note: If you're using anaconda and importing k2 on MacOS, you can probably fix this by setting the environment variable: export DYLD_LIBRARY_PATH=$CONDA_PREFIX/lib/python3.8/site-packages:$DYLD_LIBRARY_PATH

csukuangfj commented 2 years ago

What is your OS version? (I know it's centos, but what's the exact version?)

The conda package is built using Ubuntu 18.04. If your OS was released before 2018, I would recommend you to install k2 from source.

Aurora-6 commented 2 years ago

What is your OS version? (I know it's centos, but what's the exact version?)

The conda package is built using Ubuntu 18.04. If your OS was released before 2018, I would recommend you to install k2 from source.

Exactly on CentOS 7.6. I also tried to install k2 from source following "python3 setup.py install". But it still doesn't work and a cmake error occurs. "try re-running this command again with PATH=/usr/local/cuda/bin:$PATH" is also not helpful.

CMake Error at /root/miniconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:86 (message): FindCUDA says CUDA version is (usually determined by nvcc), but the CUDA headers say the version is 11.1. This often occurs when you set both CUDA_HOME and CUDA_NVCC_EXECUTABLE to non-standard locations, without also setting PATH to point to the correct nvcc. Perhaps, try re-running this command again with PATH=/usr/local/cuda/bin:$PATH. See above log messages for more diagnostics, and see https://github.com/pytorch/pytorch/issues/8092 for more details. Call Stack (most recent call first): /root/miniconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include) /root/miniconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package) cmake/torch.cmake:11 (find_package) CMakeLists.txt:279 (include)

Aurora-6 commented 2 years ago

What is your OS version? (I know it's centos, but what's the exact version?)

The conda package is built using Ubuntu 18.04. If your OS was released before 2018, I would recommend you to install k2 from source.

And this error...k2/csrc/version.h not found

-- Configuring incomplete, errors occurred! See also "/apdcephfs/private_xinyuema/k2_fsa/k2/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log". See also "/apdcephfs/private_xinyuema/k2_fsa/k2/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeError.log". cat: k2/csrc/version.h: No such file or directory make: *** No rule to make target `_k2'. Stop.

danpovey commented 2 years ago

Try to figure out where your CUDA is installed. The PATH thing was just an example.

csukuangfj commented 2 years ago

Please create a bash script containing the following content.

Delete the build directory, source the script you just created and run python3 setup.py install.

image

export CUDA_HOME=/path/to/your/cuda-11.1
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH

# See /ceph-fj/fangjun/py38/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Modules_CUDA_fix/upstream/FindCUDA.cmake
export CUDA_TOOLKIT_ROOT_DIR=$CUDA_HOME
export CUDA_TOOLKIT_ROOT=$CUDA_HOME
export CUDA_BIN_PATH=$CUDA_HOME
export CUDA_PATH=$CUDA_HOME
export CUDA_INC_PATH=$CUDA_HOME/targets/x86_64-linux
Aurora-6 commented 2 years ago

I deleted the k2 scripts and re-installed k2 from Github source. Run python3 setup.py install and install k2 done. "Finished processing dependencies for k2==1.17.dev20220714+cuda11.1.torch1.8.1" But when I import k2, import error still occurs...

import k2

Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/k2-1.17.dev20220714+cuda11.1.torch1.8.1-py3.8-linux-x86_64.egg/k2/init.py", line 24, in from _k2 import DeterminizeWeightPushingType ImportError: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /root/miniconda3/lib/python3.8/site-packages/_k2.cpython-38-x86_64-linux-gnu.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "", line 1, in File "/root/miniconda3/lib/python3.8/site-packages/k2-1.17.dev20220714+cuda11.1.torch1.8.1-py3.8-linux-x86_64.egg/k2/init.py", line 30, in raise ImportError( ImportError: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /root/miniconda3/lib/python3.8/site-packages/_k2.cpython-38-x86_64-linux-gnu.so) Note: If you're using anaconda and importing k2 on MacOS, you can probably fix this by setting the environment variable: export DYLD_LIBRARY_PATH=$CONDA_PREFIX/lib/python3.8/site-packages:$DYLD_LIBRARY_PATH

csukuangfj commented 2 years ago

Can you check whether you have deleted all installed files related to k2 before reinstalling it?

I deleted the k2 scripts

How did you do that? Which files have you deleted?

conda remove k2

should work if you previously installed k2 via conda install.

csukuangfj commented 2 years ago

Please remove all files/folders whose name contains k2 inside the directory /root/miniconda3/lib/python3.8/site-packages and then re-run python3 setup.py install.

wqn628 commented 2 years ago

your

What is your OS version? (I know it's centos, but what's the exact version?)

The conda package is built using Ubuntu 18.04. If your OS was released before 2018, I would recommend you to install k2 from source.

ah... after I installed k2 using Conda on centos7.9， i run the command: "python -m k2.version", the same error occur as follows: (k2) [smcf@localhost ~]$ python3 -m k2.version Traceback (most recent call last): File "/data/mnt/wangqingnan/Anaconda/envs/k2/lib/python3.8/site-packages/k2/init.py", line 24, in from _k2 import DeterminizeWeightPushingType ImportError: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /data/mnt/wangqingnan/Anaconda/envs/k2/lib/python3.8/site-packages/_k2.cpython-38-x86_64-linux-gnu.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/data/mnt/wangqingnan/Anaconda/envs/k2/lib/python3.8/runpy.py", line 185, in _run_module_as_main mod_name, mod_spec, code = _get_module_details(mod_name, _Error) File "/data/mnt/wangqingnan/Anaconda/envs/k2/lib/python3.8/runpy.py", line 111, in _get_module_details import(pkg_name) File "/data/mnt/wangqingnan/Anaconda/envs/k2/lib/python3.8/site-packages/k2/init.py", line 30, in raise ImportError( ImportError: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /data/mnt/wangqingnan/Anaconda/envs/k2/lib/python3.8/site-packages/_k2.cpython-38-x86_64-linux-gnu.so) Note: If you're using anaconda and importing k2 on MacOS, you can probably fix this by setting the environment variable: export DYLD_LIBRARY_PATH=$CONDA_PREFIX/lib/python3.8/site-packages:$DYLD_LIBRARY_PATH

my os is "Linux localhost.localdomain 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux" and ”CentOS Linux release 7.9.2009 (Core)“，as you say， i can not install k2 with conda？ thank for all apply。

csukuangfj commented 2 years ago

as you say， i can not install k2 with conda？ thank for all apply。

The conda package is built using Ubuntu 18.04, which cannot run on your OS as your OS, centos 7.9, was released in 2009, which is too old.

In your case, you can either switch to a newer OS or install k2 from source. See https://k2-fsa.github.io/k2/installation/from_source.html

wqn628 commented 2 years ago

as you say， i can not install k2 with conda？ thank for all apply。

The conda package is built using Ubuntu 18.04, which cannot run on your OS as your OS, centos 7.9, was released in 2009, which is too old.

In your case, you can either switch to a newer OS or install k2 from source. See https://k2-fsa.github.io/k2/installation/from_source.html

thank a lot. I got it

Aurora-6 commented 2 years ago

Please remove all files/folders whose name contains k2 inside the directory /root/miniconda3/lib/python3.8/site-packages and then re-run python3 setup.py install.

Sorry forget to thanks for your helps. I've been able to run the scripts of egs successfully. I found that conda install k2 can also succeed in centos system. The import error before is mainly because the version of CUDA is not 11.1.

I'd like to ask you one more question. Does current k2 support streaming training and decoding? In https://github.com/k2-fsa/icefall/tree/master/egs/aishell/ASR/pruned_transducer_stateless3 , the feature of whole audio seems as input for encoder in training stage. I'm not sure if the results shown in RESULTS.md are from streaming decoding or non-streaming. Sincerely thanks for your answers~

csukuangfj commented 2 years ago

Does current k2 support streaming training and decoding?

Yes, it does.

Please see

You can also find streaming ASR demos using pre-trained models from icefall in https://k2-fsa.github.io/sherpa/streaming_asr/index.html

Screen Shot 2022-07-22 at 4 57 46 PM
Aurora-6 commented 2 years ago

Does current k2 support streaming training and decoding?

Yes, it does.

Please see

You can also find streaming ASR demos using pre-trained models from icefall in https://k2-fsa.github.io/sherpa/streaming_asr/index.html

Screen Shot 2022-07-22 at 4 57 46 PM

ok, I get it. Are the streaming training and decoding scripts shown in the streaming branch of Icefall?

csukuangfj commented 2 years ago

ok, I get it. Are the streaming training and decoding scripts shown in the streaming branch of Icefall?

Please don't use the streaming branch at all.

Aurora-6 commented 2 years ago

ok, I get it. Are the streaming training and decoding scripts shown in the streaming branch of Icefall?

Please don't use the streaming branch at all.

So how can I train a streaming pruned transducer by k2? This part of codes has not been officially released?

pkufool commented 2 years ago

@Aurora-6 You can train a streaming model using recipes pruned_transducer_statelessX, all you need to do is specifying the related parameters, mainly dynamic-chunk-training, causal-convolution and num-left-chunks.

Take pruned_transducer_stateless2 as example:

./pruned_transducer_stateless2/train.py \
  --exp-dir pruned_transducer_stateless2/exp \
  --full-libri 1 \
  --dynamic-chunk-training 1 \
  --causal-convolution 1 \
  --short-chunk-size 20 \
  --num-left-chunks 4 \
  --max-duration 300 \
  --world-size 4 \
  --start-epoch 0 \
  --num-epochs 25

Search "Streaming" in https://github.com/k2-fsa/icefall/blob/master/egs/librispeech/ASR/RESULTS.md for current results and trained models.

Note: pruned_transducer_stateless{5,6} are not supported yet, will add soon.

Aurora-6 commented 2 years ago

@Aurora-6 You can train a streaming model using recipes pruned_transducer_statelessX, all you need to do is specifying the related parameters, mainly dynamic-chunk-training, causal-convolution and num-left-chunks.

Take pruned_transducer_stateless2 as example:

./pruned_transducer_stateless2/train.py \
  --exp-dir pruned_transducer_stateless2/exp \
  --full-libri 1 \
  --dynamic-chunk-training 1 \
  --causal-convolution 1 \
  --short-chunk-size 20 \
  --num-left-chunks 4 \
  --max-duration 300 \
  --world-size 4 \
  --start-epoch 0 \
  --num-epochs 25

Search "Streaming" in https://github.com/k2-fsa/icefall/blob/master/egs/librispeech/ASR/RESULTS.md for current results and trained models.

Note: pruned_transducer_stateless{5,6} are not supported yet, will add soon.

ok, thanks a lot！

wangtiance commented 2 years ago

Please remove all files/folders whose name contains k2 inside the directory /root/miniconda3/lib/python3.8/site-packages and then re-run python3 setup.py install.

Sorry forget to thanks for your helps. I've been able to run the scripts of egs successfully. I found that conda install k2 can also succeed in centos system. The import error before is mainly because the version of CUDA is not 11.1.

I'd like to ask you one more question. Does current k2 support streaming training and decoding? In https://github.com/k2-fsa/icefall/tree/master/egs/aishell/ASR/pruned_transducer_stateless3 , the feature of whole audio seems as input for encoder in training stage. I'm not sure if the results shown in RESULTS.md are from streaming decoding or non-streaming. Sincerely thanks for your answers~

Just curious, did you use 'conda install k2' and it worked on CentOS 7.6? So it seems that a newer CUDA version can also provide a newer glibc? Thanks.