search

snipsco / snips-nlu

Snips Python library to extract meaning from text
https://snips-nlu.readthedocs.io
Apache License 2.0
3.89k stars 513 forks source link

Crash at training #820

Closed joffreyvillard closed 5 years ago

joffreyvillard commented 5 years ago

Hi,

I'm experiencing consistent crash when training on a specific dataset (using Snips-NLU 0.19.7, but got the same results with 0.19.4), with the following error messages:

thread '<unnamed>' panicked at 'No such local time', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/chrono-0.4.6/src/offset/mod.rs:145:34
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
fatal runtime error: failed to initiate panic, error 5

I've spent hard time reducing the problem to a minimum, and here are the results of my tests:

Entities

I defined 2 custom entities in separate files:

Intent

Then, I defined only 1 intent "query" with 1 or 2 sentences. I found several cases described below:

With only 1 of these sentences, training is successful and fast (<5 seconds on my laptop)

With the 2 sentences above put together, it fails

With a slightly different datetime value for the first sentence, it works well

"in the 2nd quarter of 2015" instead of "2nd quarter 2015"

name: query
type: intent
utterances:
  - "[metric:metric](Transactions) [date-time:snips/datetime](in the 2nd quarter of 2015)"
  - "[metric:metric](Revenue) in [filter:filter](REF-2100005) [date-time:snips/datetime](in march 2015)"

With a slightly different filter value for the second sentence, it works but takes more than 30 seconds to train

"REF-210000" instead of "REF-2100005"

name: query
type: intent
utterances:
  - "[metric:metric](Transactions) [date-time:snips/datetime](2nd quarter 2015)"
  - "[metric:metric](Revenue) in [filter:filter](REF-210000) [date-time:snips/datetime](in march 2015)"

With yet another different filter value for the second sentence, it works well (<5 seconds)

"REF-2100" instead of "REF-2100005"

name: query
type: intent
utterances:
  - "[metric:metric](Transactions) [date-time:snips/datetime](2nd quarter 2015)"
  - "[metric:metric](Revenue) in [filter:filter](REF-2100) [date-time:snips/datetime](in march 2015)"

Backtrace

As instructed in the error message, I've run the training with RUST_BACKTRACE=1, here is the complete output:

thread '<unnamed>' panicked at 'No such local time', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/chrono-0.4.6/src/offset/mod.rs:145:34
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
   1: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:59
             at src/libstd/panicking.rs:197
   3: std::panicking::default_hook
             at src/libstd/panicking.rs:211
   4: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:474
   5: std::panicking::begin_panic
   6: chrono::offset::LocalResult<T>::unwrap
   7: rustling_ontology_moment::Moment<T>::add_months
   8: <rustling_ontology_moment::Moment<T> as core::ops::arith::Add<rustling_ontology_moment::period::PeriodComp>>::add
   9: <rustling_ontology_moment::interval_constraints::Cycle as rustling_ontology_moment::interval_constraints::IntervalConstraint<T>>::to_walker::{{closure}}
  10: rustling_ontology_moment::walker::Walker<V>::next
  11: rustling_ontology_moment::walker::Walker<V>::next
  12: <rustling_ontology_moment::interval_constraints::TakeTheNthAfter<T> as rustling_ontology_moment::interval_constraints::IntervalConstraint<T>>::to_walker::{{closure}}
  13: <rustling_ontology_moment::interval_constraints::Translate<T> as rustling_ontology_moment::interval_constraints::IntervalConstraint<T>>::to_walker::{{closure}}
  14: rustling_ontology_moment::walker::Walker<V>::next
  15: rustling_ontology_moment::walker::Walker<V>::next
  16: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T,I>>::from_iter
  17: <rustling_ontology_moment::interval_constraints::Translate<T> as rustling_ontology_moment::interval_constraints::IntervalConstraint<T>>::to_walker
  18: <rustling_ontology_moment::interval_constraints::TakeTheNthAfter<T> as rustling_ontology_moment::interval_constraints::IntervalConstraint<T>>::to_walker
  19: <rustling_ontology_values::context::ResolverContext as rustling_ontology_values::context::ParsingContext<rustling_ontology_values::dimension::Dimension>>::resolve
  20: <core::iter::adapters::Map<I,F> as core::iter::traits::iterator::Iterator>::fold
  21: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T,I>>::from_iter
  22: rustling::Parser<V,Feat,Extractor>::candidates
  23: rustling_ontology::Parser::parse_with_kind_order
  24: snips_nlu_parsers::builtin_entity_parser::BuiltinEntityParser::_extract_entities
  25: snips_nlu_parsers::builtin_entity_parser::BuiltinEntityParser::extract_entities
  26: snips_nlu_parsers_ffi_macros::builtin_entity_parser::extract_builtin_entity
  27: snips_nlu_parsers_ffi_macros::builtin_entity_parser::extract_builtin_entity_json
  28: snips_nlu_parsers_extract_builtin_entities_json
  29: ffi_call_unix64
  30: ffi_call
  31: _ctypes_callproc
  32: <unknown>
  33: PyObject_Call
  34: PyEval_EvalFrameEx
  35: <unknown>
  36: PyEval_EvalFrameEx
  37: <unknown>
  38: PyEval_EvalFrameEx
  39: <unknown>
  40: PyEval_EvalFrameEx
  41: <unknown>
  42: PyEval_EvalFrameEx
  43: <unknown>
  44: PyEval_EvalFrameEx
  45: <unknown>
  46: PyEval_EvalFrameEx
  47: <unknown>
  48: PyEval_EvalFrameEx
  49: <unknown>
  50: PyEval_EvalFrameEx
  51: PyEval_EvalCodeEx
  52: <unknown>
  53: PyObject_Call
  54: PyEval_EvalFrameEx
  55: <unknown>
  56: PyEval_EvalFrameEx
  57: PyEval_EvalCodeEx
  58: <unknown>
  59: PyObject_Call
  60: PyEval_EvalFrameEx
  61: <unknown>
  62: PyEval_EvalFrameEx
  63: PyEval_EvalCodeEx
  64: <unknown>
  65: PyObject_Call
  66: PyEval_EvalFrameEx
  67: <unknown>
  68: PyEval_EvalFrameEx
  69: PyEval_EvalFrameEx
  70: PyEval_EvalFrameEx
  71: PyEval_EvalFrameEx
  72: <unknown>
  73: PyEval_EvalCode
  74: <unknown>
  75: PyCFunction_Call
  76: PyEval_EvalFrameEx
  77: <unknown>
  78: PyEval_EvalFrameEx
  79: PyEval_EvalCodeEx
  80: <unknown>
  81: PyObject_Call
  82: <unknown>
  83: Py_Main
  84: main
  85: __libc_start_main
  86: _start
fatal runtime error: failed to initiate panic, error 5
adrienball commented 5 years ago

@joffreyvillard Thanks for reporting this, I'm investigating the issue.

adrienball commented 5 years ago

@joffreyvillard, This issue has been fixed in the latest release (0.20.0) of snips-nlu. This release is model breaking, meaning that the trainings generated with snips-nlu<0.20.0 cannot be loaded with this version. Instead, the NLU engines must be re-trained.

joffreyvillard commented 5 years ago

Very nice, thanks for the quick resolution (well noted for the breaking upgrade)!