elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
970 stars 24.82k forks source link

Allow indexed/runtime fields to use scaled_float #96443

Open henrikno opened 1 year ago

henrikno commented 1 year ago

Description

I'd like to use scaled_float on a runtime field/indexed scripted field. We have a calculated field (subtracting two timestamps to get a duration) that needs to be indexed to make visualizations load fast enough. But this can only be stored as a double (or long), which we found out is taking a lot of space because it can't compress doubles well. But we're storing a lot more precision than we need, we're interested in the second or maybe tenth of a second resolution, which scaled_float would handle. But looks like this is not supported (later found that it is documented on https://www.elastic.co/guide/en/elasticsearch/reference/master/number.html)

PUT test123
{
  "mappings": {
    "properties": {
      "test": {
        "type": "scaled_float",
        "scaling_factor": 100, 
        "script": {
          "source": "emit(1.0)"
        }
      }
    }
  }
{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "unknown parameter [script] on mapper [test] of type [scaled_float]"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "Failed to parse mapping: unknown parameter [script] on mapper [test] of type [scaled_float]",
    "caused_by": {
      "type": "mapper_parsing_exception",
      "reason": "unknown parameter [script] on mapper [test] of type [scaled_float]"
    }
  },
  "status": 400
}

Workarounds: We could use an ingest pipeline to calculate the duration. Then we can use scaled_float. But having the ability to do it as a runtime/indexed field is more convenient. Or we could manually multiply the value by scaling_factor in the script, store it as long, and update dashboards to calculate the real value (but might be slow).

elasticsearchmachine commented 1 year ago

Pinging @elastic/es-search (Team:Search)

elasticsearchmachine commented 3 months ago

Pinging @elastic/es-search-foundations (Team:Search Foundations)