tdnn_lstm_ctc training error of librispeech

hi, when I run the tdnn_lstm_ctc training of librispeech on Epoch 5, I got one error，please take a look,thanks log of error：

2021-09-17 13:28:10,440 INFO [train.py:450] Epoch 5, batch 8620, batch avg loss 1.0633, total avg loss: 1.1221, batch size: 39 2021-09-17 13:28:22,302 INFO [train.py:450] Epoch 5, batch 8630, batch avg loss 1.1049, total avg loss: 1.1507, batch size: 41 2021-09-17 13:28:25,554 WARNING [cut.py:1694] To perform mix, energy must be non-zero and non-negative (got 0.0). MonoCut with id "845a0a69-f758-7b6a-90d8-ba99fa1795c4" will not be mixed in. 2021-09-17 13:28:36,682 INFO [train.py:450] Epoch 5, batch 8640, batch avg loss 1.1305, total avg loss: 1.1622, batch size: 40 2021-09-17 13:28:49,311 INFO [train.py:450] Epoch 5, batch 8650, batch avg loss 1.1228, total avg loss: 1.1774, batch size: 37 [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void block:[0,0,0], thread: [32,0,0] block:[0,0,0], thread: [33,0,0] block:[0,0,0], thread: [34,0,0] block:[0,0,0], thread: [35,0,0] block:[0,0,0], thread: [36,0,0] block:[0,0,0], thread: [37,0,0] block:[0,0,0], thread: [38,0,0] block:[0,0,0], thread: [39,0,0] block:[0,0,0], thread: [0,0,0] block:[0,0,0], thread: [1,0,0] block:[0,0,0], thread: [2,0,0] block:[0,0,0], thread: [3,0,0] block:[0,0,0], thread: [4,0,0] block:[0,0,0], thread: [5,0,0] block:[0,0,0], thread: [6,0,0] block:[0,0,0], thread: [7,0,0] block:[0,0,0], thread: [8,0,0] block:[0,0,0], thread: [9,0,0] block:[0,0,0], thread: [10,0,0] block:[0,0,0], thread: [11,0,0] block:[0,0,0], thread: [12,0,0] block:[0,0,0], thread: [13,0,0] block:[0,0,0], thread: [14,0,0] block:[0,0,0], thread: [15,0,0] block:[0,0,0], thread: [16,0,0] block:[0,0,0], thread: [17,0,0] block:[0,0,0], thread: [18,0,0] block:[0,0,0], thread: [19,0,0] block:[0,0,0], thread: [20,0,0] block:[0,0,0], thread: [21,0,0] block:[0,0,0], thread: [22,0,0] block:[0,0,0], thread: [23,0,0] block:[0,0,0], thread: [24,0,0] block:[0,0,0], thread: [25,0,0] block:[0,0,0], thread: [26,0,0] block:[0,0,0], thread: [27,0,0] block:[0,0,0], thread: [28,0,0] block:[0,0,0], thread: [29,0,0] block:[0,0,0], thread: [30,0,0] block:[0,0,0], thread: [31,0,0] Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_s/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [32,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [33,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [34,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [35,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [36,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [37,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [38,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [39,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [0,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [1,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [2,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [3,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [4,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [5,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [6,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [7,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [8,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [9,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [10,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [11,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [12,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [13,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [14,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [15,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [16,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [17,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [18,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [19,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [20,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [21,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [22,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [23,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [24,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [25,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [26,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [27,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [28,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [29,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [30,0,0] Assertion Some bad things happened failed. /workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [31,0,0] Assertion Some bad things happened failed. tart || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0 nannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannan vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs nannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannan [F] /workspace/k2/k2/csrc/array.h:341:T k2::Array1::operator const [with T = int; int32_t = int] Check failed: ret == cudaSuccess (710 vs. 0) Error: device-side assert triggered. [ Stack-Trace: ] /opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/libk2_log.so(k2::internal::GetStackTrace()+0x47) [0x7f8c456419f7] /opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/libk2context.so(k2::Array1::operator const+0xeb9) [0x7f8c4593c8e9] /opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/libk2context.so(k2::Renumbering::ComputeOld2New()+0x14e) [0x7f8c459377ee] /opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/libk2context.so(k2::Renumbering::ComputeNew2Old()+0x7f8) [0x7f8c45938f68] /opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/libk2context.so(k2::MultiGraphDenseIntersect::FormatOutput(k2::Array1, k2::Array1)+0x7ec) [0x7f8c45a9e47c] /opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/libk2context.so(k2::IntersectDense(k2::Ragged&, k2::DenseFsaVec&, k2::Array1 const, float, k2::Ragged, k2::Array1, k2::Array1)+0x420) [0x7f8c45a8e900] /opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/_k2.cpython-38-x86_64-linux-gnu.so(+0x65390) [0x7f8c4bad4390] /opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/_k2.cpython-38-x86_64-linux-gnu.so(+0x1ee9e) [0x7f8c4ba8de9e] python3(PyCFunction_Call+0x58) [0x55e5b8fb72d8] python3(_PyObject_MakeTpCall+0x23c) [0x55e5b8fa6edc] python3(_PyEval_EvalFrameDefault+0x11dd) [0x55e5b902f4ad] python3(_PyEval_EvalCodeWithName+0x300) [0x55e5b8ffc760] python3(_PyFunction_Vectorcall+0x1e3) [0x55e5b8ffd593] python3(PyObject_CallObject+0x52) [0x55e5b9002982] /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so(THPFunction_apply(_object, _object)+0x8fd) [0x7f8d427dc39d] python3(PyCFunction_Call+0xe0) [0x55e5b8fb7360] python3(_PyObject_MakeTpCall+0x23c) [0x55e5b8fa6edc] python3(_PyEval_EvalFrameDefault+0x45a9) [0x55e5b9032879] python3(_PyEval_EvalCodeWithName+0x300) [0x55e5b8ffc760] python3(_PyFunction_Vectorcall+0x1e3) [0x55e5b8ffd593] python3(+0x10425f) [0x55e5b8f6725f] python3(_PyEval_EvalCodeWithName+0x300) [0x55e5b8ffc760] python3(_PyFunction_Vectorcall+0x1e3) [0x55e5b8ffd593] python3(+0x19aac9) [0x55e5b8ffdac9] python3(PyObject_Call+0x414) [0x55e5b8fa7874] python3(_PyEval_EvalFrameDefault+0x2088) [0x55e5b9030358] python3(_PyEval_EvalCodeWithName+0x300) [0x55e5b8ffc760] python3(_PyObject_Call_Prepend+0x181) [0x55e5b8ffe051] python3(+0x19b3fa) [0x55e5b8ffe3fa] python3(_PyObject_MakeTpCall+0x23c) [0x55e5b8fa6edc] python3(_PyEval_EvalFrameDefault+0x475) [0x55e5b902e745] python3(_PyEval_EvalCodeWithName+0x300) [0x55e5b8ffc760] python3(_PyFunction_Vectorcall+0x1e3) [0x55e5b8ffd593] python3(+0x103562) [0x55e5b8f66562] python3(_PyEval_EvalCodeWithName+0x300) [0x55e5b8ffc760] python3(_PyFunction_Vectorcall+0x1e3) [0x55e5b8ffd593] python3(+0x103562) [0x55e5b8f66562] python3(_PyEval_EvalCodeWithName+0x300) [0x55e5b8ffc760] python3(_PyFunction_Vectorcall+0x1e3) [0x55e5b8ffd593] python3(+0x103562) [0x55e5b8f66562] python3(_PyEval_EvalCodeWithName+0x300) [0x55e5b8ffc760] python3(_PyFunction_Vectorcall+0x1e3) [0x55e5b8ffd593] python3(+0x103562) [0x55e5b8f66562] python3(_PyFunction_Vectorcall+0x10b) [0x55e5b8ffd4bb] python3(+0x10425f) [0x55e5b8f6725f] python3(_PyEval_EvalCodeWithName+0x300) [0x55e5b8ffc760] python3(PyEval_EvalCode+0x23) [0x55e5b90914e3] python3(+0x22e584) [0x55e5b9091584] python3(+0x2547c4) [0x55e5b90b77c4] python3(+0x115620) [0x55e5b8f78620]

Traceback (most recent call last): File "./tdnn_lstm_ctc/train.py", line 616, in main() File "./tdnn_lstm_ctc/train.py", line 612, in main run(rank=0, world_size=1, args=args) File "./tdnn_lstm_ctc/train.py", line 575, in run train_one_epoch( File "./tdnn_lstm_ctc/train.py", line 424, in train_one_epoch loss = compute_loss( File "./tdnn_lstm_ctc/train.py", line 317, in compute_loss loss = k2.ctc_loss( File "/opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/k2/ctc_loss.py", line 136, in ctc_loss return m(decoding_graph, dense_fsa_vec, target_lengths) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/k2/ctc_loss.py", line 80, in forward lattice = intersect_dense(decoding_graph, dense_fsa_vec, File "/opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/k2/autograd.py", line 810, in intersect_dense _IntersectDenseFunction.apply(a_fsas, b_fsas, out_fsa, output_beam, File "/opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/k2/autograd.py", line 550, in forward ragged_arc, arc_map_a, arc_map_b = _k2.intersect_dense( RuntimeError: Some bad things happed.

k2-fsa / icefall

tdnn_lstm_ctc training error of librispeech #49