[Fleet] Remove experimental data streams feature

[nchaulet]

Description

Experimental data streams features (enabling tsdb, synthetic source) are not exposed in the UI anymore, and it could be dangerous for end user to play with those settings.

In my opinion we should consider removing that feature, the package ecosystem improved a lot and I think there is better way to test those features than enabling through Fleet API.

How we could remove that feature

We should deprecate those fields in the API, add a new model version of the saved object that remove that field and remove the related code path in the package installation and package policy service.

cc @lucabelluccini @kpollich

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

+1 from me these should be removed as part of our 9.0 breaking changes. The code paths to reconcile these settings are quite complex, and it'd remove a good amount of complexity to simply remove them. These are no longer needed as tsdb, synthetic source, etc are supported by integrations directly in GA.

[lucabelluccini]

The very tricky part here is we're observing users who enabled such experimental features in the past, they upgrade to newer versions where they're no more exposed in the UI, but kept in the underlying saved objects. It is leading to quite tough troubleshooting sessions as we cannot see those except checking the system indices manually. One of the observed problem is the inability to upgrade the packages because index templates are installed with tsds or synthetic source while they're not supported.

[nchaulet]

@kpollich it something that what could consider removing before 9.0 with a release note? It seems for me since we removed the UI that feature is not really usable for end users

[kpollich]

it something that what could consider removing before 9.0 with a release note? It seems for me since we removed the UI that feature is not really usable for end users

I'm willing to remove the code paths for toggling these features before 9.0 as there's not really a discoverable way to use this feature since we removed it from the UI.

To Luca's points above, we should probably solve the issue of users who enabled these features previously in a better way. That seems like it could be separate from removing the API support for the features though.

[lucabelluccini]

The main problem is:

• The most complex part is telling/notifying users they have the experimental features still enabled now that in recent versions we do not show them anymore.

Possible ideas:

• Automatically remove the experimental features on package upgrade
  • The advantage is it is "easy" and we would automatically clean up the experimental flags over time.
  • The disadvantage is this might be seen as a breaking change. But one could contest this point as those were never GA features (so they should not be used in production, even if some users do)
• Allow users disable those experimental flags (with a warning being irreversible) on the Integration package page in Fleet. The experimental flags will be shown ONLY if already present.
  • The advantage is it is "opt-in", so the user is in control.
  • Another advantage is we can take the opportunity to raise the deprecation warning in the UI.
  • The disadvantage is it requires code changes on our side to re-surface the flags.

[nchaulet]

@lucabelluccini What about automatically removing the datastreams experimental feature on stack upgrade? (with a release note that we are doing this) it is something that could be acceptable for user?

[lucabelluccini]

Doing it during a stack upgrade is not optimal. I'm going to share an example. Imagine the removal of the experimental settings changes some types of the data and therefore the kind of queries which can be run against the index. Users might spot search or indexing rejections and it would be tough to understand what is going on. From the user perspective, the integration was working with the experimental feature and they might have built dashboards or visualizations on top of the existing data.

I think it might be ok-ish to do automatically on stack upgrade ONLY if we give clear instructions before the upgrade. But in general users prefer having full control on when things happen and to PREPARE for the change.

Do we have telemetry to get an idea of how many users still rely on experimental features?

[kpollich]

I don't think we're going to be able to get to this for 9.0 unfortunately as the migration process to mitigate the breaking behavior for any users who have these fields set right now is quite complex, and we haven't had the bandwidth to define this further.