Open zhichaowang opened 3 years ago
@csukuangfj will make a PR to-morrow or the next day that should help the convergence, by using a teacher model. Did you run the tests in k2? (cd build; ctest or "make test")? That looks to me like a compilation/build problem.
... actually there is a flag to turn off the LM rescoring, I believe. That is a new feature, it may be buggy.
But since the model didn't converge, I doubt decoding will work.
@csukuangfj will make a PR to-morrow or the next day that should help the convergence, by using a teacher model. Did you run the tests in k2? (cd build; ctest or "make test")? That looks to me like a compilation/build problem.
I re-build k2 and the information are as follows:
cmake -DCMAKE_BUILD_TYPE=Release .. -- The CUDA compiler identification is NVIDIA 10.2.89 -- The CXX compiler identification is GNU 8.3.1 -- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc -- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc -- works -- Detecting CUDA compiler ABI info -- Detecting CUDA compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- K2_OS: CentOS Linux release 8.2.2004 (Core) -- Found Git: /usr/bin/git (found version "2.18.2") -- Looking for C++ include cxxabi.h -- Looking for C++ include cxxabi.h - found -- Looking for C++ include execinfo.h -- Looking for C++ include execinfo.h - found -- Performing Test K2_COMPILER_SUPPORTS_CXX14 -- Performing Test K2_COMPILER_SUPPORTS_CXX14 - Success -- C++ Standard version: 14 -- Autodetected CUDA architecture(s): 6.1 6.1 6.1 6.1 6.1 6.1 6.1 6.1 -- K2_COMPUTE_ARCH_FLAGS: -gencode;arch=compute_61,code=sm_61 CMake Warning at CMakeLists.txt:123 (message): arch 62/72 are not supported for now
-- Skipping arch 35
-- Skipping arch 50
-- Skipping arch 60
-- Adding arch 61
-- Skipping arch 70
-- Skipping arch 75
-- K2_COMPUTE_ARCHS: 61
-- Found Valgrind: /usr/bin
-- Found Valgrind: /usr/bin/valgrind
-- To check memory, run ctest -R <NAME> -D ExperimentalMemCheck
-- Downloading pybind11
-- pybind11 is downloaded to /k2/build/_deps/pybind11-src
-- pybind11 v2.6.0
-- Found PythonInterp: /anaconda3/bin/python (found version "3.8.5")
-- Found PythonLibs: /anaconda3/lib/libpython3.8.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Python executable: /anaconda3/bin/python
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found version "10.2")
-- Caffe2: CUDA detected: 10.2
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 10.2
-- Found CUDNN: /usr/local/cuda/lib64/libcudnn.so
-- Found cuDNN: v8.0.5 (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
-- /usr/local/cuda/lib64/libnvrtc.so shorthash is 08c4863f
-- Autodetected CUDA architecture(s): 6.1 6.1 6.1 6.1 6.1 6.1 6.1 6.1
-- Added CUDA NVCC flags for: -gencode;arch=compute_61,code=sm_61
-- Found Torch: /anaconda3/lib/python3.8/site-packages/torch/lib/libtorch.so
-- PyTorch version: 1.8.0
-- PyTorch cuda version: 10.2
-- Downloading cub
-- cub is downloaded to /k2/build/_deps/cub-src
-- Downloading moderngpu
-- moderngpu is downloaded to /k2/build/_deps/moderngpu-src
-- Downloading googletest
-- googletest is downloaded to /k2/build/_deps/googletest-src
-- googletest's binary dir is /k2/build/_deps/googletest-build
-- The C compiler identification is GNU 8.3.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Generated /k2/build/k2/csrc/version.h
-- Configuring done
-- Generating done
-- Build files have been written to: /k2/build
Did you run
make -j
before running
ctest?
The log is from the configure phase, not from the build phase.
@zhichaowang
Could you try https://github.com/k2-fsa/snowfall/pull/174? I think it helps.
@csukuangfj Ok,I'll try it.
@csukuangfj I re-installed lhotse, k2 and snowfall, the convergence porblem was solved.
The following are the results of w/o ali_model:
During decoding, I turned off the LM rescoring.
Thanks! I suppose it's unclear from this whether the alignment is helpful in terms of WER. We can turn it off when we don't have problems with convergence, but for now let's keep it in the code because it makes it easier to play with new models and not worry so much about will it converge.
On Sun, Apr 25, 2021 at 2:06 PM ZhichaoWang @.***> wrote:
@csukuangfj https://github.com/csukuangfj I re-installed lhotse, k2 and snowfall, the convergence porblem was solved.
The following are the results of w/o ali_model: [image: image] https://user-images.githubusercontent.com/8521283/115982617-33c08380-a5cf-11eb-8982-8c1626f5877b.png
During decoding, I turned off the LM rescoring.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/k2-fsa/snowfall/issues/170#issuecomment-826264280, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO6OX2IW6E6YU4EIYEDTKOWPTANCNFSM43KAVT4A .
And please try the LM rescoring code.
On Sun, Apr 25, 2021 at 2:38 PM Daniel Povey @.***> wrote:
Thanks! I suppose it's unclear from this whether the alignment is helpful in terms of WER. We can turn it off when we don't have problems with convergence, but for now let's keep it in the code because it makes it easier to play with new models and not worry so much about will it converge.
On Sun, Apr 25, 2021 at 2:06 PM ZhichaoWang @.***> wrote:
@csukuangfj https://github.com/csukuangfj I re-installed lhotse, k2 and snowfall, the convergence porblem was solved.
The following are the results of w/o ali_model: [image: image] https://user-images.githubusercontent.com/8521283/115982617-33c08380-a5cf-11eb-8982-8c1626f5877b.png
During decoding, I turned off the LM rescoring.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/k2-fsa/snowfall/issues/170#issuecomment-826264280, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO6OX2IW6E6YU4EIYEDTKOWPTANCNFSM43KAVT4A .
@danpovey
Add the results of LM rescoring:
The LM rescoring related configuration: --use-lm-rescoring=True --num-path=100 --max-duration=200 --output-beam-size=20
I didn't use the whole lattice for LM rescoring because of the OOM problem, even when I reduced max-duration to 10 and output-beam-size to 3. The GPU used for decoding is Tesla P40 with 22GB memory.
even when I reduced max-duration to 10 and output-beam-size to 3
Could you try the latest snowfall? I just fixed the bug about output-beam-size
in https://github.com/k2-fsa/snowfall/pull/178. Sorry for the inconvenience. The command line option --output-beam-size was never used.
I didn't use the whole lattice for LM rescoring
Rescoring with the whole lattice should give you a WER less than 6% on test-clean.
even when I reduced max-duration to 10 and output-beam-size to 3
Could you try the latest snowfall? I just fixed the bug about
output-beam-size
in #178. Sorry for the inconvenience. The command line option --output-beam-size was never used.
I tried the latest snowfall, the output-beam-size option worked. The WER on test-clean increase from 6.31% to 6.47% when -num-path=100 --output-beam-size=20. The OOM problem still exists when rescoring with the whole lattice, with --max-duration=10 --output-beam-size=1.
The OOM problem still exists
Perhaps your GPU has a very limited RAM. How about decreasing the output_beam_size further? It can be a floating-point number, e.g, 0.5, 0.1, ...
The OOM problem still exists
Perhaps your GPU has a very limited RAM. How about decreasing the output_beam_size further? It can be a floating-point number, e.g, 0.5, 0.1, ...
I found the "--output-beam-size" option has nothing to do with the OOM problem. The OOM problem occured before searching process, following is the log. The GPU capacity is 22GB.
CUDA_VISIBLE_DEVICES=7 ./mmi_att_transformer_decode.py --epoch=10 --avg=5 --use-lm-rescoring=True --num-path=-1 --max-duration=10 --output-beam-size=0.01
2021-04-26 11:35:34,393 DEBUG [mmi_att_transformer_decode.py:246] output_beam_size: 0.01
2021-04-26 11:35:34,640 DEBUG [mmi_att_transformer_decode.py:259] About to load model
2021-04-26 11:35:34,847 INFO [common.py:111] average over checkpoints ['exp-conformer-noam-mmi-att-musan-sa/epoch-5.pt', 'exp-conformer-noam-mmi-att-musan-sa/epoch-6.pt', 'exp-conformer-noam-mmi-att-musan-sa/epoch-7.pt', 'exp-conformer-noam-mmi-att-musan-sa/epoch-8.pt', 'exp-conformer-noam-mmi-att-musan-sa/epoch-9.pt']
2021-04-26 11:35:38,789 DEBUG [mmi_att_transformer_decode.py:322] Loading pre-compiled HLG
2021-04-26 11:35:43,492 INFO [mmi_att_transformer_decode.py:328] Rescoring with the whole lattice
2021-04-26 11:35:43,516 DEBUG [mmi_att_transformer_decode.py:347] Loading pre-compiled G_4_gram.pt
Traceback (most recent call last):
File "./mmi_att_transformer_decode.py", line 402, in
The OOM problem occured before searching process, following is the log.
One workaround is to use CPU to do k2.arc_sort
:
G = k2.arc_sort(G.to('cpu')).to(device)
But I am not sure whether you will encounter CUDA OOM in the later stages.
Hi guys, I have installed the latest lhotse, k2 and snowfall and run the 100h librispeech example using mmi_att_transformer_train.py. The model seems does not converge well as the valid objf no longer drops after the second epoch. And during decoding, there is an error. Following are the training log and decoding error: