search

k2-fsa / icefall

https://k2-fsa.github.io/icefall/
Apache License 2.0
844 stars 274 forks source link

How to intersect with H and LG #673

Open Temicia opened 1 year ago

Temicia commented 1 year ago

I try to intersect output lattice (H) and LG and obtained the result using the 'intersect_device' function.

But, sometimes it outputs empty hyp.

segment: 0
text:

Can you give any advice?

csukuangfj commented 1 year ago

Please have a look at https://github.com/k2-fsa/icefall/blob/2f43e4508b9eada64a9b89a4576935fb0b72694c/egs/librispeech/ASR/local/compile_hlg.py#L125

Temicia commented 1 year ago

Thanks for replying. However, the same problem arises.

csukuangfj commented 1 year ago

Please build a small H and LG so that you can visualize them and analyze the result.

Temicia commented 1 year ago

There are times when the final state does not exist when the lattice(H) and LG are combined.

compose_H,LG.pdf

csukuangfj commented 1 year ago

You can use k2.connect(HLG) to trim it.

There are times when the final state does not exist

What do your H and LG look like? Also, are you using compile_hlg.py from icefall?

Temicia commented 1 year ago

No, I don't use compile_hlg.py I used the fast beam search result of the rnnt model as H.

As follows

H = fast_beam_search(
        model=model,
        encoder_out=encoder_out,
        processed_lens=processed_lens,
        rnnt_decoding_config=rnnt_decoding_config,
        rnnt_decoding_streams_list=rnnt_decoding_streams_list,
        temperature=temperature,
    )
lattice = k2.compose(H, LG, inner_labels="tokens")

H H.pdf

The result is empty when I use 'k2.connect' because there is no final state.

pkufool commented 1 year ago

What's the purpose of composing fast_beam_search lattice with LG? You can use an LG graph in fast_beam_search directlly.

Temicia commented 1 year ago

Since fast_beam_search LG takes a long time to infer online, so I combined the lattice and LG.

pkufool commented 1 year ago

Since fast_beam_search LG takes a long time to infer online, so I combined the lattice and LG.

How large is your LG graph? From our previous benchmark fast_beam_search_LG is only slight slower than fast_beam_search.

Temicia commented 1 year ago

The size of the LG graph is about 1.1G. When online recognition was performed using Sherpa, hypothesis prediction was delayed due to slow speed.

pkufool commented 1 year ago

Ok, 1.1G is not so large. Did you use multiple threads? I think the fast_beam_search decoding methods in Sherpa do not support multiple threads very well because of the GIL in python.

pkufool commented 1 year ago

I think the fast_beam_search decoding methods in Sherpa do not support multiple threads very well because of the GIL in python.

We will fix this soon.

Temicia commented 1 year ago

Thank you for your kind response.