[Fleet]: Custom logs data gets removed from Data Streams tab on upgrading from 7.17.21>8.14.0.

[amolnater-qasource]

Kibana Build details:

VERSION: 8.14.0 BC3
BUILD: 73762
COMMIT: 2a492e1625f24336f3259b2b8df62b2b18127e81

Preconditions:

1. 7.17.21 Kibana cloud environment should be available.
2. Windows, Linux and MAC agent should be installed.

Steps to reproduce:

1. Observe custom logs data available under Data Streams tab.
2. Now upgrade 7.17.21>8.14.0
3. Observe on successful upgrade data for custom logs integration gets removed under Data Streams tab.
4. Observe no new data is getting generated under Data Streams tab, however data is available under Discover tab.

Expected Result: Custom logs data should not get removed from Data Streams tab on upgrading from 7.17.21>8.14.0.

Screen Recording: Pre-Upgrade:

Post Upgrade:

https://github.com/elastic/kibana/assets/77374876/01383826-f987-4fa2-ba48-4b6f5792917d

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Secondary review for this ticket is Done

[juliaElastic]

I could reproduce the issue, the data stream is still there after the upgrade, only not returned by Fleet's data stream API. I'll have a look why Fleet API doesn't return it.

I found that the Fleet API shows only data streams where the _meta.managed_by field is set to fleet or ingest-manager. For the data stream created by Custom logs in 7.17.21, there is no such field, that's why it's not showing up on Fleet UI. https://github.com/elastic/kibana/blob/f3a04a24ddc836d20c0d15423ce3caaf6ef4f883/x-pack/plugins/fleet/server/routes/data_streams/handlers.ts#L65

GET _data_stream/logs-mac-generic-mac

{
  "data_streams": [
    {
      "name": "logs-mac-generic-mac",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
        ...}
      ],
      "generation": 1,
      "_meta": {
        "description": "default logs template installed by x-pack",
        "managed": true
      },
      "template": "logs",
     ...

Comparing to elastic-agent datastreams, which have managed_by field:

    {
      "name": "logs-elastic_agent-default",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
       ...
      ],
      "generation": 2,
      "_meta": {
        "package": {
          "name": "elastic_agent"
        },
        "managed_by": "ingest-manager",
        "managed": true
      },
      "template": "logs-elastic_agent",

The managed_by field comes from the index template, in case of Custom Logs it is the generic logs index template which doesn't have this field. The Custom Logs package didn't have any specific index templates in 7.17 (logs-1.1.2).

I'm not sure if we can fix this by adding the managed_by field to the generic logs index template in 7.17, or if there is a better way to recognize this data stream in /data_streams API.

I think the impact of this issue can be decreased, as there is no data loss, only a UI issue.

[jen-huang]

I'll also add that the reason that these data streams show up in 7.17 even without the managed_by field is because in 7.17 we were not filtering on that field: https://github.com/elastic/kibana/blob/7.17/x-pack/plugins/fleet/server/routes/data_streams/handlers.ts#L65

If the user upgrades their Custom Logs package after upgrading the stack to 8.14, I believe their data streams will start showing up here because we do start adding managed_by to the mappings:

@kpollich I agree with Julia that's it's not a data loss issue. I also don't think it's worth fixing as an issue because upgrading all the components should resolve it.

[kpollich]

Closing as won't fix per above

[amolnater-qasource]

Hi @jen-huang

If the user upgrades their Custom Logs package after upgrading the stack to 8.14, I believe their data streams will start showing up here because we do start adding managed_by to the mappings:

We have revalidated this issue at our end and upgraded the existing custom logs integration after upgrade.

• We observed that custom logs aren't available under Data Streams tab, however new data is generated under Discover tab.

Screen Recording:

https://github.com/elastic/kibana/assets/77374876/0b423a6e-9f16-46d9-952b-54dd4b95ed9e

Please let us know if we are missing anything here.

cc: @kpollich Thanks!

[jen-huang]

We have revalidated this issue at our end and upgraded the existing custom logs integration after upgrade.

• We observed that custom logs aren't available under Data Streams tab, however new data is generated under Discover tab.

Did you upgrade the integration policies that contain the linux-generic input?

[amolnater-qasource]

@jen-huang Yes, all the integrations in the policy are updated to latest version including Custom Logs integration, still no data for linux-generic under Data Streams.

Screen Recording:

https://github.com/elastic/kibana/assets/77374876/8b0f2910-6e89-4e7c-9372-260a0ed801c2

Please let us know if we are missing anything here.

Thanks!

[jen-huang]

@amolnater-qasource Thanks for checking. I'll look into this more when I get a bit of time. Stay tuned :)

[amolnater-qasource]

Sure @jen-huang Thank you, hence we are reopening this issue.

[amolnater-qasource]

Hi Team,

We have revalidated this issue on latest 8.15.0 BC6 kibana cloud environment and found it still reproducible.

Observations:

• Custom logs data gets removed from Data Streams tab on upgrading from 7.17.23>8.15.0.

Screen Recording:

https://github.com/user-attachments/assets/4e92cd90-9c9d-4c86-9766-a7d45c28d2d0

https://github.com/user-attachments/assets/8a7ccb4f-719a-48e1-ac49-5c2778e62769

Build details: VERSION: 8.15.0 BC6 BUILD: 76360 COMMIT: 8aa0b59da12c996e3048d887546667ee6e15c7f

Please let us know if anything else is required from our end.

Thanks!