missing data: field 'usage_system', metric 'avg', error while training

[bandeep2000]

Getting this error while running command, could you please help.

Getting error:

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T01:25:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T01:30:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T01:35:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T01:40:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T01:45:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T01:50:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T01:55:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T02:00:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T02:05:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T02:10:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T02:15:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T02:20:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T02:25:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T02:30:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T02:35:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T02:40:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T02:45:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T02:55:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T03:00:00Z

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T03:05:00Z

Influx db measurement details:

select * from cpu limit 1;

name: cpu

time cpu host usage_guest usage_guest_nice usage_idle usage_iowait usage_irq usage_nice usage_softirq usage_steal usage_system usage_user

---

2018-12-17T23:43:08Z cpu-total telegraf 0 0 98.73577749685724 0.2528445006322268 0 0 0.0632111251580567 0 0.695322376738635 0.2528445006322717

Models File Details:

{

"offset": "30s",

"features": {

  "io": [

      {

          "metric": "avg",

          "field": "usage_system",

          "name": "mean_usage_system",

          "default": 0,

          "measurement": "cpu",

          "anomaly_type": "low_high"

      }

  ]

},

"bucket_interval": "5m",

"min_threshold": 75,

"max_threshold": 95,

"name": "avg_cpu_idle",

"default_datasource": "favorite_source",

"interval": "1m",

"max_evals": 5,

"type": "timeseries",

"span": 10,

"forecast": 5,

"seasonality": {

  "daytime": true,

  "weekday": true

}

}

Training command detail:

loudml train --from "now-1d" --to "now" avg_cpu_idle

[udaynaik]

I am also seeing this error - but instead of using command line, I am using LoudML Docker setup in the UI. Pulled the latest and ran docker-compose up - same error.

[regel]

I believe we need the complete log. Warnings say that some buckets are empty. They will automatically be filled with zeros as requested in the feature definition. However if all buckets are empty it is more likely that loudml is not finding the data in the TSDB. How is favorite_source defined in your config.yml?

Too many missing data points in the training date range will cause training to fail, using release 1.4.x

The coming 1.5 implementation is more robust to handle missing data points.

[ronansalmon]

LoudML version: loudml/community:1.4.3 I have the same issue using CLI or GUI (through chronograf). Datasource is influxdb over here and we don't have any data in default retention policy "autogen". We have multiple retention policies and it seams that loudml is always looking for data in autogen retention policy. Is there a way of telling loudml what retention policies to use ? Thanks,

[regel]

@ronansalmon @udaynaik @bandeep2000 yes, 1.4 and 1.5 (new OSS release, breaking changes see the release notes) have this option and it's controlled in your config.yml file.

https://loudml.io/guide/en/loudml/reference/current/_setup_your_config_yml_file.html

[pahowart]

Can we get an update on this? @regal , link you provided is broken. Release notes for 1.4 , 1.42 and 1.5 make no reference to option for retention policy in config.yml.

ty

[pahowart]

Found my answer "retention_policy:"

[regel]

INFO:root:missing data: field 'usage_system', metric 'avg', bucket: 2018-12-24T01:25:00Z

These logs are warning. They are not critical unless you see too many of them and they cause training to fail. This log line warns you that it's not possible to calculate the average for certain timestamps. Two common root causes:

1. no data for certain time interval. They are easily ignored, and you can set a default setting in your model file to force a specific value when this happens.
2. You see a lot of these warnings. Perhaps the TSDB query did not return any data at all? In this case, log the query received (or sent to) the TSDB to investigate.

@pahowart : Yes, retention_policy should help. Sorry that it's missing in the documentation. I'm adding it now. Thanks Paul for catching this.

[regel]

Documentation updated