[Fleet] `getBulkAssets` echoes whatever it receives

[consulthys]

Kibana version: 8.15.0

Elasticsearch version: 8.15.0

Server OS version: ESS

Browser version: Version 126.0.6478.185 (official build) (x86_64)

Browser OS version: MacOS 11.2.3

Original install method (e.g. download page, yum, from source, etc.): ESS

Describe the bug: The "Ingest pipelines" link available in Stack Monitoring stopped working as designed in 8.15.0.

If the elasticsearch integration assets have not been previously installed, clicking on that link redirects the user to a 404 page (Screenshot 1) instead of showing a popup inviting the user to install the assets (Screenshot 2)

Up to 8.14.3, getBulkAssets would not return an asset if it didn't exist as a saved object. Since 8.15.0, it now simply echoes back links for whatever asset is sent to it, whether fictitious or not . Since the response from that call is used as a condition to show the popup (screenshot 2) and the call always returns a non-empty response, regardless of the existence of the underlying asset, the popup is never shown.

Steps to reproduce:

1. Spin up a 8.15.0 stack (8.9.0 through 8.14.3 work fine)
2. Enable Stack Monitoring (on-prem) / Enable Logs & Metrics (in ESS)
3. Go to the Stack Monitoring application
4. Click on Overview
5. Click on the "Ingest Pipelines" link
6. Witness that you're redirected to the Dashboards application with a 404 message (screenshot 1)

Expected behavior: The expected behavior is to see a popup (screenshot 2) inviting the user to install the elasticsearch integration assets if those haven't been installed prior.

Screenshots (if relevant):

Screenshot 1: Dashboard 404

Screenshot 2: Popup inviting the user to install integration assets

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

On any version below 8.15.0, the following request returns an empty response:

POST kbn:/api/fleet/epm/bulk_assets
{
  "assetIds": [
    {
      "id": "elasticsearch-metrics-ingest-pipelines",
      "type": "dashboard"
    }
  ]
}

=> 
[]

As of 8.15.0, the same request returns whatever is passed:

POST kbn:/api/fleet/epm/bulk_assets
{
  "assetIds": [
    {
      "id": "elasticsearch-metrics-ingest-pipelines",
      "type": "dashboard"
    }
  ]
}
=>
{
  "items": [
    {
      "id": "elasticsearch-metrics-ingest-pipelines",
      "type": "dashboard",
      "attributes": {},
      "appLink": "/app/dashboards#/view/elasticsearch-metrics-ingest-pipelines"
    }
  ]
}

POST kbn:/api/fleet/epm/bulk_assets
{
  "assetIds": [
    {
      "id": "foo-bar",
      "type": "dashboard"
    }
  ]
}
=>
{
  "items": [
    {
      "id": "foo-bar",
      "type": "dashboard",
      "attributes": {},
      "appLink": "/app/dashboards#/view/foo-bar"
    }
  ]
}

Additional context:

Possibly related to https://github.com/elastic/kibana/pull/182180 as it seems to be the only one between 8.9.0 and 8.15.0 that modified the getBulkAssets function in a meaningful way.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[consulthys]

This is still happening in 8.16.0, though it is now not showing the 404 page anymore, but a different one, which isn't more helpful to users. You can see in the URL that the user was redirected to the elasticsearch-metrics-ingest-pipelines dashboard, which doesn't exist, so it's not clear on what to do next.

@kpollich @criamico Is there any chance to get this fixed? Thanks 🙏

[rseldner]

I do still see a 404 on a brand new 8.16.0 install.

Edit: for posterity, workaround would be to manually install the Elasticsearch integration's assets which will provide the elasticsearch-metrics-ingest-pipelines saved object.

POST kbn:/api/fleet/epm/packages/elasticsearch

[kpollich]

I'll bump the priority on this and schedule it for a sprint of ours in December, but we're a little low on bandwidth to get to this quickly right now. If there's interest in getting our team to provide guidance + review on a PR from the stack monitoring team or elsewhere that would likely result in this landing more quickly.

[consulthys]

We (Stack Monitoring) would love to help, but our Fleet knowledge is pretty thin 😇

[kpollich]

I'll try to provide some helpful hints about how we could implement a fix for this below. Let me know if you think this is enough to get started on contributing here. We'd really appreciate the help with this one 🙏

The getBulkAssets API handler lives here:

https://github.com/elastic/kibana/blob/1454a75986d7ddf4fa4414de399ba29aac50fa6e/x-pack/plugins/fleet/server/routes/epm/handlers.ts#L257-L280

This is essentially a thin wrapper for this "service" method in Fleet's data layer:

https://github.com/elastic/kibana/blob/1454a75986d7ddf4fa4414de399ba29aac50fa6e/x-pack/plugins/fleet/server/services/epm/packages/get_bulk_assets.ts#L44-L98

It looks like there's an explicit swallowing of errors from the getInAppUrl method here that might be overly broad, causing us to ignore 404 errors when resolving assets, e.g.

https://github.com/elastic/kibana/blob/1454a75986d7ddf4fa4414de399ba29aac50fa6e/x-pack/plugins/fleet/server/services/epm/packages/get_bulk_assets.ts#L68-L71

This is probably the first thing I see here that seems like it could be a culprit, and it looks like we also have some potentially naive logic for generating an app link that doesn't actually check if the link is valid (because the above error is squashed), e.g.

https://github.com/elastic/kibana/blob/1454a75986d7ddf4fa4414de399ba29aac50fa6e/x-pack/plugins/fleet/server/services/epm/packages/get_bulk_assets.ts#L79-L84

I think the fix for this would be to narrow the scope of this error squashing logic so we're not ignoring legitimate 404's for nonsensical assets, only space access errors.