Open kbramhendra opened 1 year ago
Is a segment an utterance?
How long is your segment?
Are you able to compare the nnet_output
of the segment between batch decoding and decoding alone?
I think you can first try using a larger beam
and max-active-states
. And do you use the latest k2? There are some discussion here https://github.com/k2-fsa/icefall/issues/292 not sure whether they are the same issue (i.e about the prunning).
Thank you for replying. The segment is an utterance, its happening with different types of durations...the nnet outputs are same in both the case...this is problem is happening intermittently, beam size 20 and max-active-states is 7000 , yes my issue is similar to 292 but i am using release post that issue. I am using 1.22 dev2022nov19 release.
I suspect this issue also relates to this https://github.com/k2-fsa/k2/pull/1087, @glynpu @danpovey I think we should revisit that PR (i.e. intersect_dense_pruned with allow parital) and merge it.
For your reference the issue is happening at k2.intersect_dense_pruned. Here i am attaching the lattice (generated by using torch.save(lattice.as_dict(), 'lattice.pt') ). Thanks
Hi...any suggestion on how to solve this ?
You can see that there is no final state in the graph (be pruned ?), so the lattice will be empty after connectting.
I think https://github.com/k2-fsa/k2/pull/1087 can solve this, can you try cherring pick the changes and see if it fix your problem.
Could you post the output of
python3 -m k2.version
?
If you are using an old version of k2, please consider updating it.
Hi for illegal memory output
_collecting environment information...
k2 version: 1.23.4 Build type: Release Git SHA1: Git date: Cuda used to build k2: 10.2 cuDNN used to build k2: 8.7.0 Python version used to build k2: 3.9 OS used to build k2: Ubuntu 18.04.6 LTS CMake version: 3.22.1 GCC version: 12.1.0 CMAKE_CUDA_FLAGS: -lineinfo --expt-extended-lambda -use_fast_math -Xptxas=-w --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -DONNX_NAMESPACE=onnx_c2 -gencode arch=compute_70,code=sm_70 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=integer_sign_change,--diag_suppress=useless_using_declaration,--diag_suppress=set_but_not_used,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=implicit_return_from_non_void_function,--diag_suppress=unsigned_compare_with_zero,--diag_suppress=declared_but_not_referenced,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda -D_GLIBCXX_USE_CXX11_ABI=0 --compiler-options -Wall --compiler-options -Wno-strict-overflow --compiler-options -Wno-unknown-pragmas CMAKE_CXX_FLAGS: -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/ubuntu/users/srikanth/envs/dist_env/envs/dist_env/include -D_GLIBCXX_USE_CXX11_ABI=0 -Wno-unused-variable -Wno-strict-overflow PyTorch version used to build k2: 1.12.0+cu102 PyTorch is using Cuda: 10.2 NVTX enabled: True With CUDA: True Disable debug: True Sync kernels : True Disable checks: False Max cpu memory allocate: 214748364800 bytes (or 200.0 GB) k2 abort: False file: /home/ubuntu/users/srikanth/envs/dist_env/envs/dist_env/lib/python3.9/site-packages/k2-1.23.4.dev20230705+cuda10.2.torch1.12.0-py3.9-linux-x86_64.egg/k2/version/version.py _k2.file: /home/ubuntu/users/srikanth/envs/dist_env/envs/dist_env/lib/python3.9/site-packages/k2-1.23.4.dev20230705+cuda10.2.torch1.12.0-py3.9-linux-x86_64.egg/_k2.cpython-39-x8664-linux-gnu.so
With the latest version I ran into some problems...so I tried to install this 1.23.4 which just previous to that. with this and changes in the #1087 am getting this problem. Earlier also i was using 1.23.4 only just with the changes #1087 i am getting this " Check failed: e == cudaSuccess (700 vs. 0) Error: an illegal memory access was encountered " this error
@kbramhendra OK, I will try to merge #1087 as soon as possible.
Thank you for the help. It would be great if you could do that.
For you reference this is the script i have post integration of #1087. Thank you
@kbramhendra Can you try https://github.com/k2-fsa/k2/pull/1218 , I made some fixes to #1087 , see if it can solve your problem. It still has some problems for chunk by chunk decoding, so I can't merge it now, sorry!
Thanks a lot of your effort and time. I really appreciate it. I have tested it on few samples its seems to be working. I will get back to if i have face any problem. But its great help. Thank you. @pkufool
hi, While I am decoding with HLG, There is some random ness to the outputs, sometimes completes segment in the batch is missing and returning empty output, but when I am decoding alone the segment its coming out properly, can you please help to decode the error ?