search

OpenVoiceOS / ovos-padatious-pipeline-plugin

A neural network intent parser
http://padatious.readthedocs.io
Apache License 2.0
0 stars 0 forks source link

random training failure #12

Closed JarbasAl closed 1 week ago

JarbasAl commented 1 week ago

ocasionally happens in logs, cant reproduce on demand

exception calling callback for <Future at 0x70a8486b9cd0 state=finished raised RuntimeError>
Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 340, in _invoke_callbacks
    callback(self)
  File "/home/miro/PycharmProjects/ovos-core/.venv/lib/python3.11/site-packages/pyee/_executor.py", line 56, in _callback
    self.emit('error', exc)
  File "/home/miro/PycharmProjects/ovos-core/.venv/lib/python3.11/site-packages/pyee/_base.py", line 116, in emit
    self._emit_handle_potential_error(event, args[0] if args else None)
  File "/home/miro/PycharmProjects/ovos-core/.venv/lib/python3.11/site-packages/pyee/_base.py", line 86, in _emit_handle_potential_error
    raise error
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miro/PycharmProjects/ovos-padatious-pipeline-plugin/ovos_padatious/opm.py", line 217, in register_intent
    self._register_object(message, 'intent', self.containers[lang].add_intent)
  File "/home/miro/PycharmProjects/ovos-padatious-pipeline-plugin/ovos_padatious/opm.py", line 205, in _register_object
    self.wait_and_train()
  File "/home/miro/PycharmProjects/ovos-padatious-pipeline-plugin/ovos_padatious/opm.py", line 149, in wait_and_train
    self.train()
  File "/home/miro/PycharmProjects/ovos-padatious-pipeline-plugin/ovos_padatious/opm.py", line 137, in train
    self.containers[lang].train(single_thread=single_thread)
  File "/home/miro/PycharmProjects/ovos-padatious-pipeline-plugin/ovos_padatious/intent_container.py", line 252, in train
    self.train_thread.join(timeout)
  File "/usr/lib/python3.11/threading.py", line 1114, in join
    raise RuntimeError("cannot join thread before it is started")
RuntimeError: cannot join thread before it is started
JarbasAl commented 1 week ago

probably related

Oct 16 14:38:00 x270.home.lan ovos-core[185179]: Exception in thread Thread-122 (train):
Oct 16 14:38:00 x270.home.lan ovos-core[185179]: Traceback (most recent call last):
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:   File "/usr/lib64/python3.11/threading.py", line 1045, in _bootstrap_inner
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:     self.run()
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:   File "/usr/lib64/python3.11/threading.py", line 982, in run
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:     self._target(*self._args, **self._kwargs)
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:   File "/home/goldyfruit/.venvs/ovos/lib64/python3.11/site-packages/ovos_padatious/training_manager.py", line 112, in train
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:     self.cls.from_file(
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:   File "/home/goldyfruit/.venvs/ovos/lib64/python3.11/site-packages/ovos_padatious/intent.py", line 67, in from_file
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:     tokens = json.load(f)
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:              ^^^^^^^^^^^^
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:   File "/usr/lib64/python3.11/json/__init__.py", line 293, in load
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:     return loads(fp.read(),
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:            ^^^^^^^^^^^^^^^^
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:   File "/usr/lib64/python3.11/json/__init__.py", line 346, in loads
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:     return _default_decoder.decode(s)
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:   File "/usr/lib64/python3.11/json/decoder.py", line 337, in decode
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:   File "/usr/lib64/python3.11/json/decoder.py", line 355, in raw_decode
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:     raise JSONDecodeError("Expecting value", s, err.value) from None
Oct 16 14:38:00 x270.home.lan ovos-core[185179]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Oct 16 14:38:00 x270.home.lan ovos-core[185179]: Exception in thread Thread-137 (train):
Oct 16 14:38:00 x270.home.lan ovos-core[185179]: Traceback (most recent call last):
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:   File "/usr/lib64/python3.11/threading.py", line 1045, in _bootstrap_inner
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:     self.run()
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:   File "/usr/lib64/python3.11/threading.py", line 982, in run
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:     self._target(*self._args, **self._kwargs)
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:   File "/home/goldyfruit/.venvs/ovos/lib64/python3.11/site-packages/ovos_padatious/training_manager.py", line 95, in train
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:     train(i)
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:   File "/home/goldyfruit/.venvs/ovos/lib64/python3.11/site-packages/ovos_padatious/training_manager.py", line 26, in _train_and_save
Oct 16 14:38:00 x270.home.lan ovos-core[185179]:     obj.train(data)
Oct 16 14:38:01 x270.home.lan ovos-core[185179]:   File "/home/goldyfruit/.venvs/ovos/lib64/python3.11/site-packages/ovos_padatious/intent.py", line 79, in train
Oct 16 14:38:01 x270.home.lan ovos-core[185179]:     i.train(train_data)
Oct 16 14:38:01 x270.home.lan ovos-core[185179]:   File "/home/goldyfruit/.venvs/ovos/lib64/python3.11/site-packages/ovos_padatious/pos_intent.py", line 88, in train
Oct 16 14:38:01 x270.home.lan ovos-core[185179]:     i.train(train_data)
Oct 16 14:38:01 x270.home.lan ovos-core[185179]:   File "/home/goldyfruit/.venvs/ovos/lib64/python3.11/site-packages/ovos_padatious/entity_edge.py", line 112, in train
Oct 16 14:38:01 x270.home.lan ovos-core[185179]:     add_sents(train_data.other_sents(self.intent_name), lambda x: 0.0)
Oct 16 14:38:01 x270.home.lan ovos-core[185179]:   File "/home/goldyfruit/.venvs/ovos/lib64/python3.11/site-packages/ovos_padatious/entity_edge.py", line 103, in add_sents
Oct 16 14:38:01 x270.home.lan ovos-core[185179]:     for sent in sents:
Oct 16 14:38:01 x270.home.lan ovos-core[185179]:   File "/home/goldyfruit/.venvs/ovos/lib64/python3.11/site-packages/ovos_padatious/train_data.py", line 50, in other_sents
Oct 16 14:38:01 x270.home.lan ovos-core[185179]:     for name, sents in self.sent_lists.items():
Oct 16 14:38:01 x270.home.lan ovos-core[185179]: RuntimeError: dictionary changed size during iteration