search

kpu / kenlm

KenLM: Faster and Smaller Language Model Queries
http://kheafield.com/code/kenlm/
Other
2.5k stars 513 forks source link

Are there different states or every model uses general purpose lm::model::State? #347

Open Malkovsky opened 3 years ago

Malkovsky commented 3 years ago

According to https://github.com/kpu/kenlm/blob/master/lm/virtual_interface.hh#L71 https://github.com/kpu/kenlm/blob/master/lm/facade.hh#L14 Every model should have its own type of state, but from https://github.com/kpu/kenlm/blob/master/lm/model.hh#L30 https://github.com/kpu/kenlm/blob/master/lm/model.hh#L136 it looks like every one of them uses general state type https://github.com/kpu/kenlm/blob/master/lm/state.hh#L15 which makes trie-based models not that effective.

I'm wondering whether there are different type of states or no?