tmp_log.txt error

[benfoley]

Need better handling of this error. It occurs

Exception in thread Thread-43:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/elpis/elpis/wrappers/objects/model.py", line 239, in background_train_task
    train()
  File "/elpis/elpis/wrappers/objects/model.py", line 232, in train
    p = run(f"cd {local_kaldi_path}; ./run.sh > {tmp_log_path}")
  File "/elpis/elpis/wrappers/objects/command.py", line 16, in run
    stderr=subprocess.STDOUT
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['bash', '-c', 'cd /elpis/state/models/ab6a2222936e2ad98bcc8232e8620674/kaldi; ./run.sh > /elpis/state/tmp_log.txt']' returned non-zero exit status 1.

Hard to reproduce, but the few times I have seen it the tmp_log.txt file ends after the aligning data line:

steps/train_deltas.sh: training pass 20
steps/train_deltas.sh: aligning data

[shuttle1987]

Just as a FYI you can pipe data with subprocess from the Python side.

[benfoley]

Should be fixed now with #121 .

[benfoley]

Still an issue with the ESPnet engine.

espnet/objects/model.py has p = run(f"cd {local_espnet_path}; ./run.sh --nj 1 &> {run_log_path}") which gives the non-zero exit status 1 error when run.sh fails for whatever reason.

Would be good to catch the errors and send a message to the GUI that the process has stopped. Otherwise the GUI will spin forever.

@mattchrlw suggest using capture_output=true in subprocess.run.

[shuttle1987]

Just by the way you can set the current working directory on the python side and that will make the ./run.sh command cleaner

[shuttle1987]

You could do something like this:

import os
import contextlib
from pathlib import Path

@contextlib.contextmanager
def working_directory(path):
    """Changes working directory and returns to previous on exit."""
    prev_cwd = Path.cwd()
    os.chdir(path)
    try:
        yield
    finally:
        os.chdir(prev_cwd)

Or if you are using subprocess you could explicitly set the cwd parameter there. I think either of these are better than building the path used for the commands via string manipulations for a number of reasons.

[benfoley]

after logging refactor this should no longer be an issue