Process killed during decoding

[bmanczak]

Question

I am running the decoding recipe for the mls model in Dutch (wav2letter/recipes/mls/decode/dutch.cfg). My process gets killed upon the construction of the language model.

I've read in #213 that this might be the memory issue to be mitigated with lower number of threads. However, as far as I know the recipe does not have any flags that would allow for changing the number of threads. How would you go about solving it?

Additional Context

I am using the default config wav2letter/recipes/mls/decode/dutch.cfg. Here is the minimal code example that yields the error:

import os
import signal
from subprocess import Popen, PIPE  

def read_current_output(process):
    while True:
        output = process.stderr.readline()
        if output.decode().strip() != "":
                print(output.decode().strip())
        if "Waiting the input in the format" in output.decode():
          break;

def create_process(cmd):
    process = Popen([cmd],
                    stdin=PIPE, stdout=PIPE, stderr=PIPE,
                    shell=True, preexec_fn=os.setsid) 
    read_current_output(process)
    return process

inference_cmd = """./flashlight/build/bin/asr/fl_asr_tutorial_inference_ctc \
  --am_path=am.bin \
  --tokens_path=tokens.txt \
  --lexicon_path=joint_lexicon.txt \
  --lm_path=mls_lm_dutch/5-gram_lm.arpa \
  --logtostderr=true \
  --sample_rate=16000 \
  --beam_size=50 \
  --beam_size_token=30 \
  --beam_threshold=100 \
  --lm_weight=1.5 \
  --word_score=0 """
inference_process = create_process(inference_cmd)

Here is the log that shows the error:

I0311 13:03:18.513072    18 InferenceCTC.cpp:89] [Inference tutorial for CTC] Reading acoustic model from am.bin
I0311 13:03:25.391456    18 InferenceCTC.cpp:140] [Inference tutorial for CTC] Network is loaded.
I0311 13:03:27.876339    18 InferenceCTC.cpp:152] [Inference tutorial for CTC] Number of classes/tokens in the network: 46
I0311 13:03:27.876513    18 InferenceCTC.cpp:155] [Inference tutorial for CTC] Number of words in the lexicon: 635421
Loading the LM will be faster if you build a binary file.
Reading mls_lm_dutch/5-gram_lm.arpa
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
********Killed

Thanks!

[tlikhomanenko]

Could you check and post arpa file size and your CPU Memory? Are you using colab or your own machine?

[bmanczak]

I've resolved the problem by converting the arpa file to bin using this functionality.

[tlikhomanenko]

I believe the problem that machine has very small CPU memory and arpa is large, while bin data are packed better. Happy to hear that you resolve the issue.