search

k2-fsa / k2

FSA/FST algorithms, differentiable, with PyTorch compatibility.
https://k2-fsa.github.io/k2
Apache License 2.0
1.1k stars 214 forks source link

Missing complete Segment output #1216

Open kbramhendra opened 1 year ago

kbramhendra commented 1 year ago

hi, While I am decoding with HLG, There is some random ness to the outputs, sometimes completes segment in the batch is missing and returning empty output, but when I am decoding alone the segment its coming out properly, can you please help to decode the error ?

csukuangfj commented 1 year ago

Is a segment an utterance?

How long is your segment?

Are you able to compare the nnet_output of the segment between batch decoding and decoding alone?

pkufool commented 1 year ago

I think you can first try using a larger beam and max-active-states. And do you use the latest k2? There are some discussion here https://github.com/k2-fsa/icefall/issues/292 not sure whether they are the same issue (i.e about the prunning).

kbramhendra commented 1 year ago

Thank you for replying. The segment is an utterance, its happening with different types of durations...the nnet outputs are same in both the case...this is problem is happening intermittently, beam size 20 and max-active-states is 7000 , yes my issue is similar to 292 but i am using release post that issue. I am using 1.22 dev2022nov19 release.

pkufool commented 1 year ago

I suspect this issue also relates to this https://github.com/k2-fsa/k2/pull/1087, @glynpu @danpovey I think we should revisit that PR (i.e. intersect_dense_pruned with allow parital) and merge it.

kbramhendra commented 1 year ago

For your reference the issue is happening at k2.intersect_dense_pruned. Here i am attaching the lattice (generated by using torch.save(lattice.as_dict(), 'lattice.pt') ). Thanks

lattices.pt.zip

kbramhendra commented 1 year ago

Hi...any suggestion on how to solve this ?

pkufool commented 1 year ago

lattice_2.pdf

You can see that there is no final state in the graph (be pruned ?), so the lattice will be empty after connectting.

I think https://github.com/k2-fsa/k2/pull/1087 can solve this, can you try cherring pick the changes and see if it fix your problem.

csukuangfj commented 1 year ago

Could you post the output of

python3 -m k2.version

?

If you are using an old version of k2, please consider updating it.

kbramhendra commented 1 year ago

Hi for illegal memory output

_collecting environment information...

k2 version: 1.23.4 Build type: Release Git SHA1: Git date: Cuda used to build k2: 10.2 cuDNN used to build k2: 8.7.0 Python version used to build k2: 3.9 OS used to build k2: Ubuntu 18.04.6 LTS CMake version: 3.22.1 GCC version: 12.1.0 CMAKE_CUDA_FLAGS: -lineinfo --expt-extended-lambda -use_fast_math -Xptxas=-w --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -DONNX_NAMESPACE=onnx_c2 -gencode arch=compute_70,code=sm_70 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=integer_sign_change,--diag_suppress=useless_using_declaration,--diag_suppress=set_but_not_used,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=implicit_return_from_non_void_function,--diag_suppress=unsigned_compare_with_zero,--diag_suppress=declared_but_not_referenced,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda -D_GLIBCXX_USE_CXX11_ABI=0 --compiler-options -Wall --compiler-options -Wno-strict-overflow --compiler-options -Wno-unknown-pragmas CMAKE_CXX_FLAGS: -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/ubuntu/users/srikanth/envs/dist_env/envs/dist_env/include -D_GLIBCXX_USE_CXX11_ABI=0 -Wno-unused-variable -Wno-strict-overflow PyTorch version used to build k2: 1.12.0+cu102 PyTorch is using Cuda: 10.2 NVTX enabled: True With CUDA: True Disable debug: True Sync kernels : True Disable checks: False Max cpu memory allocate: 214748364800 bytes (or 200.0 GB) k2 abort: False file: /home/ubuntu/users/srikanth/envs/dist_env/envs/dist_env/lib/python3.9/site-packages/k2-1.23.4.dev20230705+cuda10.2.torch1.12.0-py3.9-linux-x86_64.egg/k2/version/version.py _k2.file: /home/ubuntu/users/srikanth/envs/dist_env/envs/dist_env/lib/python3.9/site-packages/k2-1.23.4.dev20230705+cuda10.2.torch1.12.0-py3.9-linux-x86_64.egg/_k2.cpython-39-x8664-linux-gnu.so

kbramhendra commented 1 year ago

With the latest version I ran into some problems...so I tried to install this 1.23.4 which just previous to that. with this and changes in the #1087 am getting this problem. Earlier also i was using 1.23.4 only just with the changes #1087 i am getting this " Check failed: e == cudaSuccess (700 vs. 0) Error: an illegal memory access was encountered " this error

pkufool commented 1 year ago

@kbramhendra OK, I will try to merge #1087 as soon as possible.

kbramhendra commented 1 year ago

Thank you for the help. It would be great if you could do that.

kbramhendra commented 1 year ago

For you reference this is the script i have post integration of #1087. Thank you

intersect_dense_pruned.cu.zip

pkufool commented 1 year ago

@kbramhendra Can you try https://github.com/k2-fsa/k2/pull/1218 , I made some fixes to #1087 , see if it can solve your problem. It still has some problems for chunk by chunk decoding, so I can't merge it now, sorry!

kbramhendra commented 1 year ago

Thanks a lot of your effort and time. I really appreciate it. I have tested it on few samples its seems to be working. I will get back to if i have face any problem. But its great help. Thank you. @pkufool