`Could not find reference for filter-index-pattern-0` when opening Lens-based dashboard created via import API

chingis-elastic commented 2 years ago

Kibana version: 7.15.0

Elasticsearch version: 7.15.0

Server OS version: https://cloud.elastic.co

Browser version: Firefox 93.0 (64-bit)

Browser OS version: macOS 11.6

Original install method (e.g. download page, yum, from source, etc.): https://cloud.elastic.co

Describe the bug:

We have recently moved Index Patterns configuration across our stack from Saved Objects API to Index Patterns API. Now, Lens-based dashboards that we manage through Kibana import API no longer work for us. The error message in browser console is

Could not find reference for filter-index-pattern-0

while there are no other errors in Kibana logs.

Steps to reproduce:

Create index and its index pattern


PUT my-test-index/_doc/1
{
"message": "hello",
"@timestamp": "2021-10-14T11:53:30.353Z"
}

POST /api/index_patterns/index_pattern { "index_pattern": { "id": "my-test-", "timeFieldName": "@timestamp", "title": "my-test-" }, "override": true, "refresh_fields": true }

2. Create visualization in `Visualize library` based on this data:

<img width="1668" alt="Screenshot 2021-10-15 at 18 43 55" src="https://user-images.githubusercontent.com/54804012/137451542-88f77bf6-7a69-4fb4-afa3-4571eb028851.png">

<img width="1672" alt="Screenshot 2021-10-15 at 18 44 30" src="https://user-images.githubusercontent.com/54804012/137451582-7fa43ad1-7efa-461b-a4f8-30d692f31931.png">

3. Create dashboard based on this Visualization (Create Dashboard -> Add from Library)

<img width="1670" alt="Screenshot 2021-10-15 at 18 45 35" src="https://user-images.githubusercontent.com/54804012/137451630-6ccc243c-829e-4a76-842d-21b859bbf0b7.png">

<img width="1671" alt="Screenshot 2021-10-15 at 18 46 08" src="https://user-images.githubusercontent.com/54804012/137451652-12832b9c-69eb-45ea-a0ed-d4191a3594b8.png">

4. Get dashboard source via Saved Object export:

<img width="1674" alt="Screenshot 2021-10-15 at 18 50 42" src="https://user-images.githubusercontent.com/54804012/137451772-adbd67cc-c152-4fd9-b485-098a009a5886.png">

5. Post this dashboard via Kibana import API

<details>

Request to https://xxx.eu-west-1.aws.qa.cld.elstc.co:9243/api/kibana/dashboards/import?force=true

POST /api/kibana/dashboards/import?force=true HTTP/1.1 Host: xxx.eu-west-1.aws.qa.cld.elstc.co:9243 Authorization: Basic Content-Type: application/json Kbn-Xsrf: true

{ "objects": [ { "id": "bc926ea0-2d8b-11ec-ba1d-e9d8c77757b0", "type": "lens", "references": [ { "id": "my-test-", "name": "indexpattern-datasource-current-indexpattern", "type": "index-pattern" }, { "id": "my-test-", "name": "indexpattern-datasource-layer-ad38d6a1-fb4f-4a8a-a6a6-d572762f741c", "type": "index-pattern" }, { "id": "my-test-*", "name": "filter-index-pattern-0", "type": "index-pattern" } ], "attributes": { "description": "", "state": { "datasourceStates": { "indexpattern": { "layers": { "ad38d6a1-fb4f-4a8a-a6a6-d572762f741c": { "columnOrder": [ "582dac4f-feb2-4a1c-a1de-f7265b5c6ba6", "dd7cd6c0-b4e6-4211-822a-69f7ad67b093" ], "columns": { "582dac4f-feb2-4a1c-a1de-f7265b5c6ba6": { "dataType": "string", "isBucketed": true, "label": "Top values of message.keyword", "operationType": "terms", "params": { "missingBucket": false, "orderBy": { "columnId": "dd7cd6c0-b4e6-4211-822a-69f7ad67b093", "type": "column" }, "orderDirection": "desc", "otherBucket": true, "size": 5 }, "scale": "ordinal", "sourceField": "message.keyword" }, "dd7cd6c0-b4e6-4211-822a-69f7ad67b093": { "dataType": "number", "isBucketed": false, "label": "Count of records", "operationType": "count", "scale": "ratio", "sourceField": "Records" } }, "incompleteColumns": {} } } } }, "filters": [ { "$state": { "store": "appState" }, "exists": { "field": "message" }, "meta": { "alias": null, "disabled": false, "indexRefName": "filter-index-pattern-0", "key": "message", "negate": false, "type": "exists" } } ], "query": { "language": "kuery", "query": "" }, "visualization": { "axisTitlesVisibilitySettings": { "x": true, "yLeft": true, "yRight": true }, "fittingFunction": "None", "gridlinesVisibilitySettings": { "x": true, "yLeft": true, "yRight": true }, "labelsOrientation": { "x": 0, "yLeft": 0, "yRight": 0 }, "layers": [ { "accessors": [ "dd7cd6c0-b4e6-4211-822a-69f7ad67b093" ], "layerId": "ad38d6a1-fb4f-4a8a-a6a6-d572762f741c", "layerType": "data", "position": "top", "seriesType": "bar_stacked", "showGridlines": false, "xAccessor": "582dac4f-feb2-4a1c-a1de-f7265b5c6ba6" } ], "legend": { "isVisible": true, "position": "right" }, "preferredSeriesType": "bar_stacked", "tickLabelsVisibilitySettings": { "x": true, "yLeft": true, "yRight": true }, "valueLabels": "hide", "yLeftExtent": { "mode": "full" }, "yRightExtent": { "mode": "full" } } }, "title": "My viz in Visualize Library", "visualizationType": "lnsXY" } }, { "id": "f6dcc3d0-2d8b-11ec-ba1d-e9d8c77757b0", "type": "dashboard", "references": [ { "id": "bc926ea0-2d8b-11ec-ba1d-e9d8c77757b0", "name": "8b87b3ba-3810-4ac3-9787-60dbde360b4d:panel_8b87b3ba-3810-4ac3-9787-60dbde360b4d", "type": "lens" } ], "attributes": { "description": "", "hits": 0, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}" }, "optionsJSON": "{\"useMargins\":true,\"syncColors\":false,\"hidePanelTitles\":false}", "panelsJSON": "[{\"version\":\"7.15.0\",\"type\":\"lens\",\"gridData\":{\"x\":0,\"y\":0,\"w\":24,\"h\":15,\"i\":\"8b87b3ba-3810-4ac3-9787-60dbde360b4d\"},\"panelIndex\":\"8b87b3ba-3810-4ac3-9787-60dbde360b4d\",\"embeddableConfig\":{\"enhancements\":{}},\"panelRefName\":\"panel_8b87b3ba-3810-4ac3-9787-60dbde360b4d\"}]", "timeRestore": false, "title": "My dash with viz from Visualize Library", "version": 1 } } ] } < HTTP/2.0 200 OK < Cache-Control: private, no-cache, no-store, must-revalidate < Content-Type: application/json; charset=utf-8 < Date: Fri, 15 Oct 2021 07:53:31 GMT < Kbn-License-Sig: xxx < Kbn-Name: xxx < Referrer-Policy: no-referrer-when-downgrade < Vary: accept-encoding < X-Cloud-Request-Id: oVmeGUKkQMSoAOzBOa1jfA < X-Content-Type-Options: nosniff < X-Found-Handling-Cluster: xxx < X-Found-Handling-Instance: instance-0000000017


</details>

6. Try to open dashboard in UI, get empty screen and `Could not find reference for filter-index-pattern-0` error message in browser console:

<img width="1677" alt="Screenshot 2021-10-15 at 18 55 15" src="https://user-images.githubusercontent.com/54804012/137452391-bb437088-31c7-4fdf-94b2-5a71f9784d03.png">

**Expected behavior:**

**Screenshots (if relevant):**

**Errors in browser console (if relevant):**



**Provide logs and/or server output (if relevant):** No significant logs on server side, but I can provide them at request.

**Any additional context:**

The whole use case works without any issues if Visuliazation was created as part of Dashboard creation flow - e.g. it wasn't saved in Visualize Library.

elasticmachine commented 2 years ago

Pinging @elastic/kibana-vis-editors (Team:VisEditors)

flash1293 commented 2 years ago

Please note that this seems to use the deprecated dashboard import api: https://www.elastic.co/guide/en/kibana/current/dashboard-api.html

not sure which team owns it but I guess it’s a problem on that level.

mbondyra commented 2 years ago

Hey @chingis-elastic, when I repeat the steps from the issue it works fine for me. I noticed that my saved object has some extra properties that don't exist in the POST body user provided:

     "coreMigrationVersion": "7.15.0",
      "migrationVersion": {
          "lens": "7.15.0"
       },

Could you ask the customer if they removed it manually or how are they process saved object to dashboard import api? I agree with Joe though that it might be better for them to use https://www.elastic.co/guide/en/kibana/current/saved-objects-api-import.html instead of deprecated dashboard api.

Extra questions (thanks @flash1293) Also, how exactly do they export saved objects? Via UI or are you using an api? If yes which one? Could you provide an example of how exactly the saved object looks like after exporting?

chingis-elastic commented 2 years ago

@mbondyra Thanks, good catch! I removed handling of those 2 fields in our code some time ago naively thinking it wouldn't impact anything - most dashboards seemed to work (except Lens ones, now that we see). Putting MigrationVersion and CoreMigrationVersion back resolves the issue. So yeah, totally on our side - I'll add the fix on Monday.

The reason I removed them is that I wanted our payloads be "version agnostic", e.g. whenever a user pulls a dashboard after minor edit, they wouldn't get a local diff related to version change. Now I see that users will have to live with those minor version-related diffs.

@flash1293 At the moment, we use /saved_objects/_bulk_get endpoint to get dashboard(s) and all their references. Later, we import those via /dashboards/import endpoint as shown in the description.

flash1293 commented 2 years ago

The version information is important here because saved objects frequently change their internal format in small and subtle ways and this version information makes sure we can internally run migrations on them before storing them so they continue to work.

About the dashboards import api - I don’t know the exact plans here but this api will definitely go away at some point, it makes sense to start thinking about how to switch to the saved objects import api linked above.

closing this as it’s resolved - thanks for reaching out!

elastic / kibana

`Could not find reference for filter-index-pattern-0` when opening Lens-based dashboard created via import API #115149