search

elastic / package-spec

EPR package specifications
Other
17 stars 70 forks source link

[Proposal] Consolidate use of ecs.version field #737

Open jsoriano opened 4 months ago

jsoriano commented 4 months ago

We have different sources of ECS fields, and different places where ecs.version can be set. This proposal attempts to define some explicit rules about the expectations of the ecs.version field.

The different sources of ECS fields definitions are, at least:

The proposal would be focused on making the final documents to have the greater known ECS version affecting the package, with two main tasks:

cc @ishleenk17 @zmoog @elastic/ecosystem @kpollich

mrodm commented 4 months ago

Is this issue https://github.com/elastic/elastic-package/issues/1066 similar to the first task mentioned in the description ? @jsoriano

[elastic-package] If a version is set in _dev/build/build.yml, elastic-package system tests must check that ecs.version is set to this version or greater. Package developers may need to add pipeline processors to satisfy this check.

jsoriano commented 4 months ago

Yes! Good finding, I will link this issue here. Thanks.

ishleenk17 commented 4 months ago

Thank you for creating the issue. This would be a pre-requisite for having a complete solution for adopting ecs@mappings for Integrations.

ishleenk17 commented 4 months ago
  • [elastic-package] If a version is set in _dev/build/build.yml, elastic-package system tests must check that ecs.version is set to this version or greater. Package developers may need to add pipeline processors to satisfy this check.

Would this mean that we will not be getting rid of build.yml ? In case we are using ecs@mappings, we will not need this.

But if we have to overwrite some of the ECS fields (say in case of TSDB), then we might need build.yml?

jsoriano commented 4 months ago

Would this mean that we will not be getting rid of build.yml ? In case we are using ecs@mappings, we will not need this.

By now build.yml is going to stay in case we need a escape hatch to explicitly import some field. It will be also needed for packages supporting versions of the stack older than 8.13.

But if we have to overwrite some of the ECS fields (say in case of TSDB), then we might need build.yml?

Correct, build.yml is still helpful on this case.

flash1293 commented 4 months ago

About the test in elastic-package - nice addition, not sure about the priority as it won't take affect on recent versions of the stack anyway (are we automatically running tests on older stack versions?)

About the second part - makes a lot of sense to me, we need to make sure the fleet-provided pipeline processor runs after the package-provided pipeline so we can "upgrade" an outdated ecs.version property.

ishleenk17 commented 4 months ago

About the second part - makes a lot of sense to me, we need to make sure the fleet-provided pipeline processor runs after the package-provided pipeline so we can "upgrade" an outdated ecs.version property.

If we can set it in the Fleet final pipeline, this can be achieved.

  • The value used must be the max version between the version of the stack and the version found in the document, if any.

@jsoriano : Say, stack version is 8.14, then the max would be taken as that. But our latest ecs.version is 8.11. So I think we will not be able to directly take the max of stack or document. Since ecs mappings version and stack version is not the same.

jsoriano commented 4 months ago

our latest ecs.version is 8.11. So I think we will not be able to directly take the max of stack or document. Since ecs mappings version and stack version is not the same.

Interesting, I thought ECS version was more coupled to the stack version. Fleet or some other component of the stack will need to know then what is the version being supported by ecs@mappings.

flash1293 commented 4 months ago

@felixbarny do you know how the fleet plugin in Kibana could detect which ecs version ecs@mappings corresponds to?

felixbarny commented 4 months ago

The ECS version of ecs@mappings is always the same as the stack version.

flash1293 commented 4 months ago

@felixbarny As @ishleenk17 noted above, the last technical release of https://github.com/elastic/ecs is 8.11, but I guess what you are saying is that if there isn't an explicit ecs release that just means that the new ecs version is identical to the previous one?

Two examples: