[Prometheus] Mixed histogram values results in document rejection

VCCPlindsten commented 4 months ago

With use_types: true documents that contain histogram values such as 0.20500000000000002 (Not exactly representable by a float32) will get rejected with the following error in metricbeat:

{"type":"illegal_argument_exception","reason":"mapper [prometheus.controller_runtime_reconcile_time_seconds.histogram.values] cannot be changed from type [float] to [long]"}

Observed in remote_write metricset, may also exist in the other metricsets.

Workaround

Set a @custom component template (e.g. metrics-prometheus.remote_write@custom) with mappings that set the type as double.

This dynamic template snippet maps all floats as double, which may or may not be desired for all use-cases

[
  {
    "floats": {
      "match_mapping_type": "double",
      "mapping": {
        "type": "double"
      }
    }
  }
]

Proposed solution

Make histogram values mappings explicitly double in the integration.

VCCPlindsten commented 4 months ago

Further investigation reveals the original cause is not what it at first seemed, instead the problem comes from attempting to dynamically map the histogram .values array when it has a "mixed" payload (both things detected as integers and things detected as floats).

This only seems to happen on the first document ingested, when one has been successfully ingested, the type is set, and the values are coerced/cast to the mapped type.

Updated workaround

Dynamic template

      {
        "prometheus.*.histogram.values": {
          "path_match": "prometheus.*.histogram.values",
          "mapping": {
            "type": "double"
          },
          "match_mapping_type": "*"
        }
      }

VCCPlindsten commented 3 months ago

This turns out to be a 8.12 regression - and the problems encountered here are in fact a failure of applying templates in correct order. Resolved by https://github.com/elastic/kibana/pull/175970 in a somewhat non-obvious way. Fixed in 8.12.1

elastic / integrations