elastic / ecs

Elastic Common Schema
https://www.elastic.co/what-is/ecs
Apache License 2.0
1.01k stars 418 forks source link

Allow use of dash character (`-`) in `data_stream.namespace` #2388

Open ebeahan opened 1 month ago

ebeahan commented 1 month ago

Summary

Propose ECS removes the restriction of using the dash character, -, in the data_stream.namespace field.

Motivation:

RFC 0009 - Data stream fields includes this restrictions for the data_stream.namespace field:

No longer than 100 chars

The current description for data_stream.namespace also restricts using the dash character:

    - name: namespace
      level: extended
      type: constant_keyword
      example: production
      description: >
        A user defined namespace. Namespaces are useful to allow grouping of data.

        Many users already organize their indices this way, and the data stream naming scheme now provides this
        best practice as a default. Many users will populate this field with `default`. If no value is used, it falls back to `default`.

        Beyond the Elasticsearch index naming criteria noted above, `namespace` value has the additional restrictions:
          * Must not contain `-` <= Not present in RFC
          * No longer than 100 characters
      short: A user defined namespace. Namespaces are useful to allow grouping of data.

Suspect these restrictions may have been mistakenly copied from the same restriction on data_stream.type and data_stream.dataset.

I would not consider the change breaking. It makes data_stream.namespace more permissive and the data_stream.* field set is still considered beta.

carsonip commented 1 month ago

I am on board with making data_stream.namespace more permissive in order to not introduce breaking changes in existing implementations.

However, as a counter argument, allowing - in data_stream.namespace makes it a bit tricky to come up with correct index patterns in some edge cases. I'm not sure if that's considered yet.

For example, let's say we have 2 data streams, one called metrics-generic.otel-default, another DS metrics-generic-my.otel-ns where namespace is my.otel-ns. They will both match metrics-*.otel-* index pattern, which may cause some index templates to be applied unintentionally.

cc @felixbarny

felixbarny commented 1 month ago

I don't see the risk of that being fundamentally different to metrics-*.otel-* matching datasets that happen to end with .otel (for example metrics-apm.app.myservice.otel-default) but aren't actually in the OTel native format.