output of exmaples provided

[vigneshmj1997]

i ran the example provided in the examples section of pyython bindings i am getting error in loading lm Loading the LM will be faster if you build a binary file. Reading /home/vignesh/wav2letter/src/decoder/test/lm.arpa ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100

---

malloc(): invalid size (unsorted) Aborted (core dumped)

THEN I TIRED MAKING CUSTOM LM(ZERO LM) the results i got were all same Showing top 5 results: score=-53.213056673295796 prediction='| t c u | ' r k | x u y w 1' score=-53.213056673295796 prediction='| t c u | ' r k | x u y w 1' score=-53.213056673295796 prediction='| t c u | ' r k | x u y w 1' score=-53.213056673295796 prediction='| t c u | ' r k | x u y w 1' score=-53.213056673295796 prediction='| t c u | ' r k | x u y w 1'

[tlikhomanenko]

The problem could be in kenlm compilation. Could you use exactly the same version we use in the docker and follow installation in the docker https://github.com/facebookresearch/wav2letter/blob/master/Dockerfile-CUDA-Base#L84 ?

For your custom ZeroLM - I need the code to see what are you doing.

[vigneshmj1997]

i am not using docker i had compiled the latest version on kenlm

class MyPyLM(LM): mapping_states = dict() # store simple additional int for each stated

def __init__(self):
    LM.__init__(self)

def start(self, start_with_nothing):
    state = LMState()
    self.mapping_states[state] = 0
    return state

def score(self, state : LMState, token_index : int):
    """
    Evaluate language model based on the current lm state and new word
    Parameters:
    -----------
    state: current lm state
    token_index: index of the word 
                 (can be lexicon index then you should store inside LM the 
                  mapping between indices of lexicon and lm, or lm index of a word)

    Returns:
    --------
    (LMState, float): pair of (new state, score for the current word)
    """
    outstate = state.child(token_index)
    if outstate not in self.mapping_states:
        self.mapping_states[outstate] = self.mapping_states[state] + 1
    return (outstate, 0)

def finish(self, state: LMState):
    """
    Evaluate eos for language model based on the current lm state

    Returns:
    --------
    (LMState, float): pair of (new state, score for the current word)
    """
    outstate = state.child(-1)
    if outstate not in self.mapping_states:
        self.mapping_states[outstate] = self.mapping_states[state] + 1
    return (outstate, 0)

this is the code lm = KenLM(os.path.join(data_path, "lm.arpa"), word_dict) i replaced above line with lm = MyPyLM() and i was getting the follwing output

THEN I TIRED MAKING CUSTOM LM(ZERO LM) the results i got were all same Showing top 5 results: score=-53.213056673295796 prediction='| t c u | ' r k | x u y w 1' score=-53.213056673295796 prediction='| t c u | ' r k | x u y w 1' score=-53.213056673295796 prediction='| t c u | ' r k | x u y w 1' score=-53.213056673295796 prediction='| t c u | ' r k | x u y w 1' score=-53.213056673295796 prediction='| t c u | ' r k | x u y w 1'

[bohdantrotsenko]

@vigneshmj1997 did you do cd kenlm && git checkout e47088ddfae810a5ee4c8a9923b5f8071bed1ae8 ?

[tlikhomanenko]

Just try to run installation like this

    cd /root && git clone https://github.com/kpu/kenlm.git && \
    cd kenlm && git checkout e47088ddfae810a5ee4c8a9923b5f8071bed1ae8 && \
    mkdir build && cd build && \
    cmake .. -DCMAKE_POSITION_INDEPENDENT_CODE=ON && \
    make -j$(nproc) && make install

and let me know if this still doesn't work for you. Regarding your zeroLM test: did you use this example https://github.com/facebookresearch/wav2letter/blob/master/bindings/python/examples/decoder_example.py and just change the kenlm to this zeroLM?