Support for field with multiple names / hard link aliases

[ruflin]

With the move to ECS and semconv, what fields to use has become more and more standardised and users are migrating to it. Part of the migration is using field aliases to point to the ECS field so the data shipper does not have be adjusted but queries against the data can be written in a standardised way. The challenge is, multiple shippers send data to the same data stream with different names for the same field.

Lets go through an example where host.name exists in 3 fields:

• host.name: The recommended ECS field, used for querying and eventually shipping data
• host_name: The field that is used by some shippers
• resource.attributes.host.name: Field that comes from some otel data

The user wants to write queries against host.name. At first, an alias is created to point host.name to host_name:

PUT alias-challenge
{
  "mappings": {
    "properties": {
      "host_name": {
        "type": "keyword"
      },
      "host.name": {
        "type": "alias",
        "path": "host_name"
      },
      "resource.host.name": {
        "type": "keyword"
      }
    }
  }
}

Now the user can query on host_name and host.name. Unfortunately the resource.host.name is not included yet because an array for aliases is not supported:

PUT alias-challenge
{
  "mappings": {
    "properties": {
      "host_name": {
        "type": "keyword"
      },
      "host.name": {
        "type": "alias",
        "path": ["host_name", "resource.host.name"]
      },
      "resource.host.name": {
        "type": "keyword"
      }
    }
  }
}

But having an array for aliases is not enough. Eventually some of the shippers migrate to use host.name which means data is sent to the alias itself:

POST alias-challenge/_doc
{
  "host.name": "elastic.co"
}

This leads to the error "reason": "[2:16] Cannot write to a field alias [host.name].". The ideal scenario would be that a single field could have multiple names or compared to the linux file system, hard links can be created. It is possible to query and ingest into all field names and the field exists until the last reference is removed.

Having support for hard links would simplify the migration to ECS / semconv for users.

Doing the standardisation in an ingest pipeline is a not a solution as also the old fields still have to be queried. One solution that sometimes is used to work around the limitation is duplicate the data into each field, but that is not simple and has a negative impact on storage.

Implementation ideas

Two ideas below on how this could look like, but I'm sure better solutions can be found.

Idea 1:

PUT alias-challenge
{
  "mappings": {
    "properties": {
      "host.name": {
        "type": "keyword",
        "hard_links": ["host_name", "resource.host.name"]
      }
    }
  }
}

Idea 2:

PUT alias-challenge
{
  "mappings": {
    "properties": {
      "host_name": {
        "type": "hard_link"
        "path": "host.name"
      },
      "host.name": {
        "type": "keyword",
      },
      "resource.host.name": {
        "type": "hard_link",
        "path": "host.name"
      }
    }
  }
}

Related links / discussions

• https://github.com/elastic/elasticsearch/issues/87969
• https://github.com/elastic/elasticsearch/issues/104744
• https://github.com/elastic/elasticsearch/issues/89093
• https://github.com/elastic/elasticsearch/issues/90345

[felixbarny]

Another idea:

PUT alias-challenge
{
  "mappings": {
    "properties": {
      "host_name": {
        "path": "keyword"
      },
      "host.name": {
        "type": "keyword",
      },
      "resource.attributes.host.name": {
        "type": "keyword"
      }
    }
  },
  "aliases": {
    ["host.name", "host_name", "resource.attributes.host.name"]
  }
}

This has the advantage that the field types don't need to be changed (which isn't possible for an existing index). The aliases section could behave similar to the runtime section in the sense that it can be dynamically changed on an existing index mapping.

While we would expect that only one of the fields is present, there's the edge case where both host.name and host_name is present on the doc. The order at which the aliases are defined then determines from which field to fetch (first one wins).

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)