Closed jingyu1220 closed 1 year ago
How large is your LG?
If it is too large, we suggest that you prune your G before composing it with L.
The G.fst.txt is 68G. As we want to use a large G to get better performance.
This error is like an array out of bounds problem
Likely the issue is that after composing with L, some of the indexes go outside the range of int32. I believe we may have some examples of using OpenFST instead of k2 to compile decoding graphs, as it has slightly larger maximum limits (although there is also an int32 limitation, just with some differences of implementation).
Wait, 68G?? You sure you don't mean 68M? 68G is way way out of bounds. Normally when people want a good LM they rely on neural language models.
I encountered a similar issue. The size of my LM is 3.7G, my dictionary size is 250000 and I have 4000 syllable tokens. When I cut the LM to 0.7G, it works. So is there any way to fix the error when the LM size is large? k2-version: 1.24.3 Git SHA1: 460c841d0329273e4d8ee60e204d450e48a78245
Error:
INFO [compile_lg.py:107] Connecting LG after k2.determinize
INFO [compile_lg.py:110] Removing disambiguation symbols on LG
/path/to/k2-exp/k2/k2/csrc/array.h:501:void k2::Array1
[ Stack-Trace: ]
/path/to/k2-exp/k2/build_debug/lib/libk2_log.so(k2::internal::GetStackTrace()+0x46) [0x7f009f8e7696]
/path/to/k2-exp/k2/build_debug/lib/libk2context.so(k2::internal::Logger::~Logger()+0x35) [0x7f009ffdd5cf]
/path/to/k2-exp/k2/build_debug/lib/libk2context.so(k2::Array1
/path/to/k2-exp/k2/build_debug/lib/libk2context.so(k2::Array1
/path/to/k2-exp/k2/build_debug/lib/libk2context.so(k2::RaggedShapeFromTotSizes(std::shared_ptr
the error is : Check failed: state_index_c + 2 == arcindexes.size() (-1909392433 vs. 2385574863)