elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.72k stars 8.14k forks source link

[maps] saved object migration to migrate tile_map and region_map visualizations to maps #105115

Closed nreese closed 2 months ago

nreese commented 3 years ago

tile_map and region_map visualizations are getting replaced with maps in 8.0. Saved object migration required to migrate tile_map and region_map visualizations to maps

download saved object export

- tile_map visualization saved object ``` { "attributes": { "description": "", "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"query\": {\"query\":\"\",\"language\":\"kuery\"},\"filter\": [],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}" }, "title": "tilemap", "uiStateJSON": "{\"mapZoom\":2,\"mapCenter\": [40.17185103580504,-123.1384584921219]}", "version": 1, "visState": "{\"title\":\"tilemap\",\"type\":\"tile_map\",\"aggs\": [{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"geohash_grid\",\"params\":{\"field\":\"geo.coordinates\",\"autoPrecision\":true,\"precision\":2,\"useGeocentroid\":true,\"isFilteredByCollar\":true},\"schema\":\"segment\"}],\"params\":{\"colorSchema\":\"Yellow to Red\",\"mapType\":\"Scaled Circle Markers\",\"isDesaturated\":true,\"addTooltip\":true,\"heatClusterSize\":1.5,\"legendPosition\":\"bottomright\",\"mapZoom\":2,\"mapCenter\":[0,0],\"wms\":{\"enabled\":false,\"url\":\"\",\"options\":{\"version\":\"\",\"layers\":\"\",\"format\":\"image/png\",\"transparent\":true,\"attribution\":\"\",\"styles\":\"\"},\"selectedTmsLayer\":{\"origin\":\"elastic_maps_service\",\"id\":\"road_map\",\"minZoom\":0,\"maxZoom\":20,\"attribution\":\"OpenStreetMap contributors | OpenMapTiles | Elastic Maps Service\"}}}}" }, "coreMigrationVersion": "8.0.0", "id": "24eefbb0-e0e0-11eb-8303-39c9ad80bd90", "migrationVersion": { "visualization": "7.14.0" }, "references": [ { "id": "90943e30-9a47-11e8-b64d-95841ca0b247", "name": "kibanaSavedObjectMeta.searchSourceJSON.index", "type": "index-pattern" } ], "type": "visualization", "updated_at": "2021-07-09T18:04:46.189Z", "version": "WzE2MDMsMV0=" } ```
- tile_map visualization saved object as map saved object ``` { "attributes": { "description": "", "layerListJSON": "[{\"sourceDescriptor\": {\"type\":\"EMS_TMS\",\"isAutoSelect\":true},\"id\":\"3e09b33a-b60d-4c9e-9d51-690410528d99\",\"label\":null,\"minZoom\":0,\"maxZoom\":24,\"alpha\":1,\"visible\":true,\"style\":{\"type\":\"TILE\"},\"includeInFitToBounds\":true,\"type\":\"VECTOR_TILE\"},{\"alpha\":0.75,\"id\":\"b9a235a0-fdd8-481f-850c-d02ba087f136\",\"includeInFitToBounds\":true,\"joins\": [],\"label\":\"tilemap\",\"maxZoom\":24,\"minZoom\":0,\"sourceDescriptor\":{\"applyGlobalQuery\":true,\"applyGlobalTime\":true,\"geoField\":\"geo.coordinates\",\"id\":\"f01deb4a-af20-4735-8be8-40fb60f96d5c\",\"metrics\":[{\"type\":\"count\"}],\"requestType\":\"point\",\"resolution\":\"MOST_FINE\",\"type\":\"ES_GEO_GRID\",\"indexPatternRefName\":\"layer_1_source_index_pattern\"},\"style\":{\"isTimeAware\":true,\"properties\":{\"fillColor\":{\"options\":{\"color\":\"Yellow to Red\",\"colorCategory\":\"palette_0\",\"field\":{\"name\":\"doc_count\",\"origin\":\"source\"},\"fieldMetaOptions\":{\"isEnabled\":false,\"sigma\":3},\"type\":\"ORDINAL\"},\"type\":\"DYNAMIC\"},\"icon\":{\"options\":{\"value\":\"marker\"},\"type\":\"STATIC\"},\"iconOrientation\":{\"options\":{\"orientation\":0},\"type\":\"STATIC\"},\"iconSize\":{\"options\":{\"field\":{\"name\":\"doc_count\",\"origin\":\"source\"},\"fieldMetaOptions\":{\"isEnabled\":false,\"sigma\":3},\"maxSize\":18,\"minSize\":7},\"type\":\"DYNAMIC\"},\"labelBorderColor\":{\"options\":{\"color\":\"#FFFFFF\"},\"type\":\"STATIC\"},\"labelBorderSize\":{\"options\":{\"size\":\"SMALL\"}},\"labelColor\":{\"options\":{\"color\":\"#000000\"},\"type\":\"STATIC\"},\"labelSize\":{\"options\":{\"size\":14},\"type\":\"STATIC\"},\"labelText\":{\"options\":{\"value\":\"\"},\"type\":\"STATIC\"},\"lineColor\":{\"options\":{\"color\":\"#3d3d3d\"},\"type\":\"STATIC\"},\"lineWidth\":{\"options\":{\"size\":1},\"type\":\"STATIC\"},\"symbolizeAs\":{\"options\":{\"value\":\"circle\"}}},\"type\":\"VECTOR\"},\"type\":\"VECTOR\",\"visible\":true}]", "mapStateJSON": "{\"zoom\":1.71,\"center\": {\"lon\":0,\"lat\":19.94277},\"timeFilters\":{\"from\":\"now-7d/d\",\"to\":\"now\"},\"refreshConfig\":{\"isPaused\":true,\"interval\":0},\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filters\": [],\"settings\":{\"autoFitToDataBounds\":false,\"backgroundColor\":\"#ffffff\",\"disableInteractive\":false,\"disableTooltipControl\":false,\"hideToolbarOverlay\":false,\"hideLayerControl\":false,\"hideViewControl\":false,\"initialLocation\":\"LAST_SAVED_LOCATION\",\"fixedLocation\":{\"lat\":0,\"lon\":0,\"zoom\":2},\"browserLocation\":{\"zoom\":2},\"maxZoom\":24,\"minZoom\":0,\"showScaleControl\":false,\"showSpatialFilters\":true,\"showTimesliderToggleButton\":true,\"spatialFiltersAlpa\":0.3,\"spatialFiltersFillColor\":\"#DA8B45\",\"spatialFiltersLineColor\":\"#DA8B45\"}}", "title": "migrated tilemap", "uiStateJSON": "{\"isLayerTOCOpen\":true,\"openTOCDetails\": []}" }, "coreMigrationVersion": "8.0.0", "id": "2b7b0230-e0e0-11eb-8303-39c9ad80bd90", "migrationVersion": { "map": "7.14.0" }, "references": [ { "id": "90943e30-9a47-11e8-b64d-95841ca0b247", "name": "layer_1_source_index_pattern", "type": "index-pattern" } ], "type": "map", "updated_at": "2021-07-09T18:04:57.173Z", "version": "WzE2MDksMV0=" } ```
- region_map visualization saved object ``` { "attributes": { "description": "", "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"query\": {\"query\":\"\",\"language\":\"kuery\"},\"filter\": [],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}" }, "title": "region map", "uiStateJSON": "{\"mapZoom\":2,\"mapCenter\": [5.983380462042426,-24.422737960659024]}", "version": 1, "visState": "{\"title\":\"region map\",\"type\":\"region_map\",\"aggs\": [{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"geo.src\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":5,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"segment\"}],\"params\":{\"legendPosition\":\"bottomright\",\"addTooltip\":true,\"colorSchema\":\"Yellow to Red\",\"emsHotLink\":\"https://maps.elastic.co/v7.13?locale=en#file/world_countries\",\"isDisplayWarning\":true,\"wms\":{\"enabled\":false,\"url\":\"\",\"options\":{\"version\":\"\",\"layers\":\"\",\"format\":\"image/png\",\"transparent\":true,\"attribution\":\"\",\"styles\":\"\"},\"selectedTmsLayer\":{\"origin\":\"elastic_maps_service\",\"id\":\"road_map\",\"minZoom\":0,\"maxZoom\":20,\"attribution\":\"OpenStreetMap contributors | OpenMapTiles | Elastic Maps Service\"}},\"mapZoom\":2,\"mapCenter\":[0,0],\"outlineWeight\":1,\"showAllShapes\":true,\"selectedLayer\":{\"name\":\"World Countries\",\"origin\":\"elastic_maps_service\",\"id\":\"world_countries\",\"created_at\":\"2020-10-28T16:16:08.720286\",\"attribution\":\"Made with NaturalEarth | © OpenStreetMap contributors | Elastic Maps Service\",\"fields\":[{\"type\":\"id\",\"name\":\"iso2\",\"description\":\"ISO 3166-1 alpha-2 code\"},{\"type\":\"id\",\"name\":\"iso3\",\"description\":\"ISO 3166-1 alpha-3 code\"},{\"type\":\"id\",\"name\":\"iso_numeric\",\"description\":\"ISO 3166-1 numeric code\"},{\"type\":\"property\",\"name\":\"name\",\"description\":\"name\"}],\"format\":\"topojson\",\"meta\":{\"feature_collection_path\":\"data\"},\"layerId\":\"elastic_maps_service.World Countries\",\"isEMS\":true},\"selectedJoinField\":{\"type\":\"id\",\"name\":\"iso2\",\"description\":\"ISO 3166-1 alpha-2 code\"}}}" }, "coreMigrationVersion": "8.0.0", "id": "faf485f0-e0df-11eb-8303-39c9ad80bd90", "migrationVersion": { "visualization": "7.14.0" }, "references": [ { "id": "90943e30-9a47-11e8-b64d-95841ca0b247", "name": "kibanaSavedObjectMeta.searchSourceJSON.index", "type": "index-pattern" } ], "type": "visualization", "updated_at": "2021-07-09T18:03:35.764Z", "version": "WzE1NjYsMV0=" } ```
- region_map visualization saved object as map saved object ``` { "attributes": { "description": "", "layerListJSON": "[{\"sourceDescriptor\": {\"type\":\"EMS_TMS\",\"isAutoSelect\":true},\"id\":\"a02586ad-24df-44d8-a1c9-f90e5eac7109\",\"label\":null,\"minZoom\":0,\"maxZoom\":24,\"alpha\":1,\"visible\":true,\"style\":{\"type\":\"TILE\"},\"includeInFitToBounds\":true,\"type\":\"VECTOR_TILE\"},{\"alpha\":0.75,\"id\":\"095b7613-3971-41ce-a029-078402aae2a0\",\"includeInFitToBounds\":true,\"joins\": [{\"leftField\":\"iso2\",\"right\":{\"applyGlobalQuery\":true,\"applyGlobalTime\":true,\"id\":\"cac8fce1-f692-4f35-b096-da11d7e08c35\",\"indexPatternTitle\":\"kibana_sample_data_logs\",\"metrics\":[{\"type\":\"count\"}],\"size\":5,\"term\":\"geo.src\",\"type\":\"ES_TERM_SOURCE\",\"indexPatternRefName\":\"layer_1_join_0_index_pattern\"}}],\"label\":\"region map\",\"maxZoom\":24,\"minZoom\":0,\"sourceDescriptor\":{\"id\":\"world_countries\",\"tooltipProperties\":[\"name\",\"iso2\"],\"type\":\"EMS_FILE\"},\"style\":{\"isTimeAware\":true,\"properties\":{\"fillColor\":{\"options\":{\"color\":\"Yellow to Red\",\"colorCategory\":\"palette_0\",\"field\":{\"name\":\"__kbnjoin__count__cac8fce1-f692-4f35-b096-da11d7e08c35\",\"origin\":\"join\"},\"fieldMetaOptions\":{\"isEnabled\":false,\"sigma\":3},\"type\":\"ORDINAL\"},\"type\":\"DYNAMIC\"},\"icon\":{\"options\":{\"value\":\"marker\"},\"type\":\"STATIC\"},\"iconOrientation\":{\"options\":{\"orientation\":0},\"type\":\"STATIC\"},\"iconSize\":{\"options\":{\"size\":6},\"type\":\"STATIC\"},\"labelBorderColor\":{\"options\":{\"color\":\"#FFFFFF\"},\"type\":\"STATIC\"},\"labelBorderSize\":{\"options\":{\"size\":\"SMALL\"}},\"labelColor\":{\"options\":{\"color\":\"#000000\"},\"type\":\"STATIC\"},\"labelSize\":{\"options\":{\"size\":14},\"type\":\"STATIC\"},\"labelText\":{\"options\":{\"value\":\"\"},\"type\":\"STATIC\"},\"lineColor\":{\"options\":{\"color\":\"#41937c\"},\"type\":\"STATIC\"},\"lineWidth\":{\"options\":{\"size\":1},\"type\":\"STATIC\"},\"symbolizeAs\":{\"options\":{\"value\":\"circle\"}}},\"type\":\"VECTOR\"},\"type\":\"VECTOR\",\"visible\":true}]", "mapStateJSON": "{\"zoom\":1.71,\"center\": {\"lon\":0,\"lat\":19.94277},\"timeFilters\":{\"from\":\"now-7d/d\",\"to\":\"now\"},\"refreshConfig\":{\"isPaused\":true,\"interval\":0},\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filters\": [],\"settings\":{\"autoFitToDataBounds\":false,\"backgroundColor\":\"#ffffff\",\"disableInteractive\":false,\"disableTooltipControl\":false,\"hideToolbarOverlay\":false,\"hideLayerControl\":false,\"hideViewControl\":false,\"initialLocation\":\"LAST_SAVED_LOCATION\",\"fixedLocation\":{\"lat\":0,\"lon\":0,\"zoom\":2},\"browserLocation\":{\"zoom\":2},\"maxZoom\":24,\"minZoom\":0,\"showScaleControl\":false,\"showSpatialFilters\":true,\"showTimesliderToggleButton\":true,\"spatialFiltersAlpa\":0.3,\"spatialFiltersFillColor\":\"#DA8B45\",\"spatialFiltersLineColor\":\"#DA8B45\"}}", "title": "migrated region map", "uiStateJSON": "{\"isLayerTOCOpen\":true,\"openTOCDetails\": []}" }, "coreMigrationVersion": "8.0.0", "id": "079bcfc0-e0e0-11eb-8303-39c9ad80bd90", "migrationVersion": { "map": "7.14.0" }, "references": [ { "id": "90943e30-9a47-11e8-b64d-95841ca0b247", "name": "layer_1_join_0_index_pattern", "type": "index-pattern" } ], "type": "map", "updated_at": "2021-07-09T18:03:56.990Z", "version": "WzE1NzQsMV0=" } ```
- dashboard saved object with by reference tile_map and region_map panels ``` { "attributes": { "description": "", "hits": 0, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"query\": {\"query\":\"\",\"language\":\"kuery\"},\"filter\": []}" }, "optionsJSON": "{\"useMargins\":true,\"syncColors\":false,\"hidePanelTitles\":false}", "panelsJSON": "[{\"version\":\"8.0.0\",\"type\":\"visualization\",\"gridData\": {\"x\":0,\"y\":0,\"w\":24,\"h\":15,\"i\":\"be583e6c-9dc9-44ed-bcde-14df1fa8eb88\"},\"panelIndex\":\"be583e6c-9dc9-44ed-bcde-14df1fa8eb88\",\"embeddableConfig\":{\"enhancements\":{}},\"panelRefName\":\"panel_be583e6c-9dc9-44ed-bcde-14df1fa8eb88\"},{\"version\":\"8.0.0\",\"type\":\"visualization\",\"gridData\":{\"x\":24,\"y\":0,\"w\":24,\"h\":15,\"i\":\"d07f657e-2fa9-4fd6-adff-4aaef2b11d6c\"},\"panelIndex\":\"d07f657e-2fa9-4fd6-adff-4aaef2b11d6c\",\"embeddableConfig\":{\"enhancements\":{}},\"panelRefName\":\"panel_d07f657e-2fa9-4fd6-adff-4aaef2b11d6c\"}]", "timeRestore": false, "title": "by ref visualizations", "version": 1 }, "coreMigrationVersion": "8.0.0", "id": "4304db60-e0e0-11eb-8303-39c9ad80bd90", "migrationVersion": { "dashboard": "7.14.0" }, "references": [ { "id": "24eefbb0-e0e0-11eb-8303-39c9ad80bd90", "name": "be583e6c-9dc9-44ed-bcde-14df1fa8eb88:panel_be583e6c-9dc9-44ed-bcde-14df1fa8eb88", "type": "visualization" }, { "id": "faf485f0-e0df-11eb-8303-39c9ad80bd90", "name": "d07f657e-2fa9-4fd6-adff-4aaef2b11d6c:panel_d07f657e-2fa9-4fd6-adff-4aaef2b11d6c", "type": "visualization" } ], "type": "dashboard", "updated_at": "2021-07-09T18:05:36.665Z", "version": "WzE2MjgsMV0=" } ```
- dashboard saved object with by reference map panels ``` { "attributes": { "description": "", "hits": 0, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"query\": {\"query\":\"\",\"language\":\"kuery\"},\"filter\": []}" }, "optionsJSON": "{\"useMargins\":true,\"syncColors\":false,\"hidePanelTitles\":false}", "panelsJSON": "[{\"version\":\"8.0.0\",\"type\":\"map\",\"gridData\": {\"x\":0,\"y\":0,\"w\":24,\"h\":15,\"i\":\"fd150bc4-1eb3-49e4-9ddb-879f4a520465\"},\"panelIndex\":\"fd150bc4-1eb3-49e4-9ddb-879f4a520465\",\"embeddableConfig\":{\"mapCenter\":{\"lat\":19.94277,\"lon\":0,\"zoom\":1.71},\"mapBuffer\":{\"minLon\":-180,\"minLat\":-66.51326,\"maxLon\":180,\"maxLat\":66.51326},\"isLayerTOCOpen\":true,\"openTOCDetails\": [],\"hiddenLayers\":[],\"enhancements\":{}},\"panelRefName\":\"panel_fd150bc4-1eb3-49e4-9ddb-879f4a520465\"},{\"version\":\"8.0.0\",\"type\":\"map\",\"gridData\":{\"x\":24,\"y\":0,\"w\":24,\"h\":15,\"i\":\"d83845f3-29fc-4179-b85a-bd9a48518ce5\"},\"panelIndex\":\"d83845f3-29fc-4179-b85a-bd9a48518ce5\",\"embeddableConfig\":{\"mapCenter\":{\"lat\":19.94277,\"lon\":0,\"zoom\":1.71},\"mapBuffer\":{\"minLon\":-180,\"minLat\":-66.51326,\"maxLon\":180,\"maxLat\":66.51326},\"isLayerTOCOpen\":true,\"openTOCDetails\":[],\"hiddenLayers\":[],\"enhancements\":{}},\"panelRefName\":\"panel_d83845f3-29fc-4179-b85a-bd9a48518ce5\"}]", "timeRestore": false, "title": "by ref maps", "version": 1 }, "coreMigrationVersion": "8.0.0", "id": "4d3de450-e0e0-11eb-8303-39c9ad80bd90", "migrationVersion": { "dashboard": "7.14.0" }, "references": [ { "id": "2b7b0230-e0e0-11eb-8303-39c9ad80bd90", "name": "fd150bc4-1eb3-49e4-9ddb-879f4a520465:panel_fd150bc4-1eb3-49e4-9ddb-879f4a520465", "type": "map" }, { "id": "079bcfc0-e0e0-11eb-8303-39c9ad80bd90", "name": "d83845f3-29fc-4179-b85a-bd9a48518ce5:panel_d83845f3-29fc-4179-b85a-bd9a48518ce5", "type": "map" } ], "type": "dashboard", "updated_at": "2021-07-09T18:05:53.815Z", "version": "WzE2NDgsMV0=" } ```
- dashboard saved object with by value tile_map and region_map panels ``` { "attributes": { "description": "", "hits": 0, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"query\": {\"query\":\"\",\"language\":\"kuery\"},\"filter\": []}" }, "optionsJSON": "{\"useMargins\":true,\"syncColors\":false,\"hidePanelTitles\":false}", "panelsJSON": "[{\"version\":\"8.0.0\",\"type\":\"visualization\",\"gridData\": {\"x\":0,\"y\":0,\"w\":24,\"h\":15,\"i\":\"cdac890e-f4cb-4d27-b1b5-8bf41e1f60ba\"},\"panelIndex\":\"cdac890e-f4cb-4d27-b1b5-8bf41e1f60ba\",\"embeddableConfig\":{\"savedVis\":{\"title\":\"tilemap\",\"description\":\"\",\"type\":\"tile_map\",\"params\":{\"colorSchema\":\"Yellow to Red\",\"mapType\":\"Scaled Circle Markers\",\"isDesaturated\":true,\"addTooltip\":true,\"heatClusterSize\":1.5,\"legendPosition\":\"bottomright\",\"mapZoom\":2,\"mapCenter\": [0,0],\"wms\":{\"enabled\":false,\"url\":\"\",\"options\":{\"version\":\"\",\"layers\":\"\",\"format\":\"image/png\",\"transparent\":true,\"attribution\":\"\",\"styles\":\"\"},\"selectedTmsLayer\":{\"origin\":\"elastic_maps_service\",\"id\":\"road_map\",\"minZoom\":0,\"maxZoom\":20,\"attribution\":\"OpenStreetMap contributors | OpenMapTiles | Elastic Maps Service\"}}},\"uiState\":{\"mapZoom\":2,\"mapCenter\":[40.17185103580504,-123.1384584921219]},\"data\":{\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"geohash_grid\",\"params\":{\"field\":\"geo.coordinates\",\"autoPrecision\":true,\"precision\":2,\"useGeocentroid\":true,\"isFilteredByCollar\":true,\"boundingBox\":{\"top_left\":{\"lat\":90,\"lon\":-180},\"bottom_right\":{\"lat\":-67.46047,\"lon\":180}}},\"schema\":\"segment\"}],\"searchSource\":{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"index\":\"90943e30-9a47-11e8-b64d-95841ca0b247\",\"filter\":[]}}},\"enhancements\":{}}},{\"version\":\"8.0.0\",\"type\":\"visualization\",\"gridData\":{\"x\":24,\"y\":0,\"w\":24,\"h\":15,\"i\":\"22423ce7-eccf-4d44-8ff2-a60a2d57fd92\"},\"panelIndex\":\"22423ce7-eccf-4d44-8ff2-a60a2d57fd92\",\"embeddableConfig\":{\"savedVis\":{\"title\":\"region map\",\"description\":\"\",\"type\":\"region_map\",\"params\":{\"legendPosition\":\"bottomright\",\"addTooltip\":true,\"colorSchema\":\"Yellow to Red\",\"emsHotLink\":\"https://maps.elastic.co/v7.13?locale=en#file/world_countries\",\"isDisplayWarning\":true,\"wms\":{\"enabled\":false,\"url\":\"\",\"options\":{\"version\":\"\",\"layers\":\"\",\"format\":\"image/png\",\"transparent\":true,\"attribution\":\"\",\"styles\":\"\"},\"selectedTmsLayer\":{\"origin\":\"elastic_maps_service\",\"id\":\"road_map\",\"minZoom\":0,\"maxZoom\":20,\"attribution\":\"OpenStreetMap contributors | OpenMapTiles | Elastic Maps Service\"}},\"mapZoom\":2,\"mapCenter\":[0,0],\"outlineWeight\":1,\"showAllShapes\":true,\"selectedLayer\":{\"name\":\"World Countries\",\"origin\":\"elastic_maps_service\",\"id\":\"world_countries\",\"created_at\":\"2020-10-28T16:16:08.720286\",\"attribution\":\"Made with NaturalEarth | © OpenStreetMap contributors | Elastic Maps Service\",\"fields\":[{\"type\":\"id\",\"name\":\"iso2\",\"description\":\"ISO 3166-1 alpha-2 code\"},{\"type\":\"id\",\"name\":\"iso3\",\"description\":\"ISO 3166-1 alpha-3 code\"},{\"type\":\"id\",\"name\":\"iso_numeric\",\"description\":\"ISO 3166-1 numeric code\"},{\"type\":\"property\",\"name\":\"name\",\"description\":\"name\"}],\"format\":\"topojson\",\"meta\":{\"feature_collection_path\":\"data\"},\"layerId\":\"elastic_maps_service.World Countries\",\"isEMS\":true},\"selectedJoinField\":{\"type\":\"id\",\"name\":\"iso2\",\"description\":\"ISO 3166-1 alpha-2 code\"}},\"uiState\":{\"mapZoom\":2,\"mapCenter\":[5.983380462042426,-24.422737960659024]},\"data\":{\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"geo.src\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":5,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"segment\"}],\"searchSource\":{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"index\":\"90943e30-9a47-11e8-b64d-95841ca0b247\",\"filter\":[]}}},\"enhancements\":{}}}]", "timeRestore": false, "title": "by value visualizations", "version": 1 }, "coreMigrationVersion": "8.0.0", "id": "61aee4c0-e0e0-11eb-8303-39c9ad80bd90", "migrationVersion": { "dashboard": "7.14.0" }, "references": [ { "id": "90943e30-9a47-11e8-b64d-95841ca0b247", "name": "cdac890e-f4cb-4d27-b1b5-8bf41e1f60ba:kibanaSavedObjectMeta.searchSourceJSON.index", "type": "index-pattern" }, { "id": "90943e30-9a47-11e8-b64d-95841ca0b247", "name": "22423ce7-eccf-4d44-8ff2-a60a2d57fd92:kibanaSavedObjectMeta.searchSourceJSON.index", "type": "index-pattern" } ], "type": "dashboard", "updated_at": "2021-07-09T18:06:28.110Z", "version": "WzE2NzksMV0=" } ```
- dashboard saved object with by reference map panels ``` { "attributes": { "description": "", "hits": 0, "kibanaSavedObjectMeta": { "searchSourceJSON": "{\"query\": {\"query\":\"\",\"language\":\"kuery\"},\"filter\": []}" }, "optionsJSON": "{\"useMargins\":true,\"syncColors\":false,\"hidePanelTitles\":false}", "panelsJSON": "[{\"version\":\"8.0.0\",\"type\":\"map\",\"gridData\": {\"x\":0,\"y\":0,\"w\":24,\"h\":15,\"i\":\"6939008d-992f-4f12-b3b0-487b499dd6d8\"},\"panelIndex\":\"6939008d-992f-4f12-b3b0-487b499dd6d8\",\"embeddableConfig\":{\"mapCenter\":{\"lat\":19.94277,\"lon\":0,\"zoom\":1.71},\"mapBuffer\":{\"minLon\":-180,\"minLat\":-66.51326,\"maxLon\":180,\"maxLat\":66.51326},\"isLayerTOCOpen\":true,\"openTOCDetails\": [],\"hiddenLayers\":[],\"enhancements\":{},\"attributes\":{\"title\":\"migrated tilemap\",\"description\":\"\",\"layerListJSON\":\"[{\\\"sourceDescriptor\\\":{\\\"type\\\":\\\"EMS_TMS\\\",\\\"isAutoSelect\\\":true},\\\"id\\\":\\\"3e09b33a-b60d-4c9e-9d51-690410528d99\\\",\\\"label\\\":null,\\\"minZoom\\\":0,\\\"maxZoom\\\":24,\\\"alpha\\\":1,\\\"visible\\\":true,\\\"style\\\":{\\\"type\\\":\\\"TILE\\\"},\\\"includeInFitToBounds\\\":true,\\\"type\\\":\\\"VECTOR_TILE\\\"},{\\\"alpha\\\":0.75,\\\"id\\\":\\\"b9a235a0-fdd8-481f-850c-d02ba087f136\\\",\\\"includeInFitToBounds\\\":true,\\\"joins\\\":[],\\\"label\\\":\\\"tilemap\\\",\\\"maxZoom\\\":24,\\\"minZoom\\\":0,\\\"sourceDescriptor\\\":{\\\"applyGlobalQuery\\\":true,\\\"applyGlobalTime\\\":true,\\\"geoField\\\":\\\"geo.coordinates\\\",\\\"id\\\":\\\"f01deb4a-af20-4735-8be8-40fb60f96d5c\\\",\\\"metrics\\\":[{\\\"type\\\":\\\"count\\\"}],\\\"requestType\\\":\\\"point\\\",\\\"resolution\\\":\\\"MOST_FINE\\\",\\\"type\\\":\\\"ES_GEO_GRID\\\",\\\"indexPatternId\\\":\\\"90943e30-9a47-11e8-b64d-95841ca0b247\\\"},\\\"style\\\":{\\\"isTimeAware\\\":true,\\\"properties\\\":{\\\"fillColor\\\":{\\\"options\\\":{\\\"color\\\":\\\"Yellow to Red\\\",\\\"colorCategory\\\":\\\"palette_0\\\",\\\"field\\\":{\\\"name\\\":\\\"doc_count\\\",\\\"origin\\\":\\\"source\\\"},\\\"fieldMetaOptions\\\":{\\\"isEnabled\\\":false,\\\"sigma\\\":3},\\\"type\\\":\\\"ORDINAL\\\"},\\\"type\\\":\\\"DYNAMIC\\\"},\\\"icon\\\":{\\\"options\\\":{\\\"value\\\":\\\"marker\\\"},\\\"type\\\":\\\"STATIC\\\"},\\\"iconOrientation\\\":{\\\"options\\\":{\\\"orientation\\\":0},\\\"type\\\":\\\"STATIC\\\"},\\\"iconSize\\\":{\\\"options\\\":{\\\"field\\\":{\\\"name\\\":\\\"doc_count\\\",\\\"origin\\\":\\\"source\\\"},\\\"fieldMetaOptions\\\":{\\\"isEnabled\\\":false,\\\"sigma\\\":3},\\\"maxSize\\\":18,\\\"minSize\\\":7},\\\"type\\\":\\\"DYNAMIC\\\"},\\\"labelBorderColor\\\":{\\\"options\\\":{\\\"color\\\":\\\"#FFFFFF\\\"},\\\"type\\\":\\\"STATIC\\\"},\\\"labelBorderSize\\\":{\\\"options\\\":{\\\"size\\\":\\\"SMALL\\\"}},\\\"labelColor\\\":{\\\"options\\\":{\\\"color\\\":\\\"#000000\\\"},\\\"type\\\":\\\"STATIC\\\"},\\\"labelSize\\\":{\\\"options\\\":{\\\"size\\\":14},\\\"type\\\":\\\"STATIC\\\"},\\\"labelText\\\":{\\\"options\\\":{\\\"value\\\":\\\"\\\"},\\\"type\\\":\\\"STATIC\\\"},\\\"lineColor\\\":{\\\"options\\\":{\\\"color\\\":\\\"#3d3d3d\\\"},\\\"type\\\":\\\"STATIC\\\"},\\\"lineWidth\\\":{\\\"options\\\":{\\\"size\\\":1},\\\"type\\\":\\\"STATIC\\\"},\\\"symbolizeAs\\\":{\\\"options\\\":{\\\"value\\\":\\\"circle\\\"}}},\\\"type\\\":\\\"VECTOR\\\"},\\\"type\\\":\\\"VECTOR\\\",\\\"visible\\\":true}]\",\"mapStateJSON\":\"{\\\"zoom\\\":1.71,\\\"center\\\":{\\\"lon\\\":0,\\\"lat\\\":19.94277},\\\"timeFilters\\\":{\\\"from\\\":\\\"now-7d/d\\\",\\\"to\\\":\\\"now\\\"},\\\"refreshConfig\\\":{\\\"isPaused\\\":true,\\\"interval\\\":0},\\\"query\\\":{\\\"query\\\":\\\"\\\",\\\"language\\\":\\\"kuery\\\"},\\\"filters\\\":[],\\\"settings\\\":{\\\"autoFitToDataBounds\\\":false,\\\"backgroundColor\\\":\\\"#ffffff\\\",\\\"disableInteractive\\\":false,\\\"disableTooltipControl\\\":false,\\\"hideToolbarOverlay\\\":false,\\\"hideLayerControl\\\":false,\\\"hideViewControl\\\":false,\\\"initialLocation\\\":\\\"LAST_SAVED_LOCATION\\\",\\\"fixedLocation\\\":{\\\"lat\\\":0,\\\"lon\\\":0,\\\"zoom\\\":2},\\\"browserLocation\\\":{\\\"zoom\\\":2},\\\"maxZoom\\\":24,\\\"minZoom\\\":0,\\\"showScaleControl\\\":false,\\\"showSpatialFilters\\\":true,\\\"showTimesliderToggleButton\\\":true,\\\"spatialFiltersAlpa\\\":0.3,\\\"spatialFiltersFillColor\\\":\\\"#DA8B45\\\",\\\"spatialFiltersLineColor\\\":\\\"#DA8B45\\\"}}\",\"uiStateJSON\":\"{\\\"isLayerTOCOpen\\\":true,\\\"openTOCDetails\\\":[]}\",\"references\":[{\"name\":\"layer_1_source_index_pattern\",\"type\":\"index-pattern\",\"id\":\"90943e30-9a47-11e8-b64d-95841ca0b247\"}]}}},{\"version\":\"8.0.0\",\"type\":\"map\",\"gridData\":{\"x\":24,\"y\":0,\"w\":24,\"h\":15,\"i\":\"87cd9c53-5e8b-4d56-89aa-47cd145e5297\"},\"panelIndex\":\"87cd9c53-5e8b-4d56-89aa-47cd145e5297\",\"embeddableConfig\":{\"mapCenter\":{\"lat\":19.94277,\"lon\":0,\"zoom\":1.71},\"mapBuffer\":{\"minLon\":-180,\"minLat\":-66.51326,\"maxLon\":180,\"maxLat\":66.51326},\"isLayerTOCOpen\":true,\"openTOCDetails\":[],\"hiddenLayers\":[],\"enhancements\":{},\"attributes\":{\"title\":\"migrated region map\",\"description\":\"\",\"layerListJSON\":\"[{\\\"sourceDescriptor\\\":{\\\"type\\\":\\\"EMS_TMS\\\",\\\"isAutoSelect\\\":true},\\\"id\\\":\\\"a02586ad-24df-44d8-a1c9-f90e5eac7109\\\",\\\"label\\\":null,\\\"minZoom\\\":0,\\\"maxZoom\\\":24,\\\"alpha\\\":1,\\\"visible\\\":true,\\\"style\\\":{\\\"type\\\":\\\"TILE\\\"},\\\"includeInFitToBounds\\\":true,\\\"type\\\":\\\"VECTOR_TILE\\\"},{\\\"alpha\\\":0.75,\\\"id\\\":\\\"095b7613-3971-41ce-a029-078402aae2a0\\\",\\\"includeInFitToBounds\\\":true,\\\"joins\\\":[{\\\"leftField\\\":\\\"iso2\\\",\\\"right\\\":{\\\"applyGlobalQuery\\\":true,\\\"applyGlobalTime\\\":true,\\\"id\\\":\\\"cac8fce1-f692-4f35-b096-da11d7e08c35\\\",\\\"indexPatternTitle\\\":\\\"kibana_sample_data_logs\\\",\\\"metrics\\\":[{\\\"type\\\":\\\"count\\\"}],\\\"size\\\":5,\\\"term\\\":\\\"geo.src\\\",\\\"type\\\":\\\"ES_TERM_SOURCE\\\",\\\"indexPatternId\\\":\\\"90943e30-9a47-11e8-b64d-95841ca0b247\\\"}}],\\\"label\\\":\\\"region map\\\",\\\"maxZoom\\\":24,\\\"minZoom\\\":0,\\\"sourceDescriptor\\\":{\\\"id\\\":\\\"world_countries\\\",\\\"tooltipProperties\\\":[\\\"name\\\",\\\"iso2\\\"],\\\"type\\\":\\\"EMS_FILE\\\"},\\\"style\\\":{\\\"isTimeAware\\\":true,\\\"properties\\\":{\\\"fillColor\\\":{\\\"options\\\":{\\\"color\\\":\\\"Yellow to Red\\\",\\\"colorCategory\\\":\\\"palette_0\\\",\\\"field\\\":{\\\"name\\\":\\\"__kbnjoin__count__cac8fce1-f692-4f35-b096-da11d7e08c35\\\",\\\"origin\\\":\\\"join\\\"},\\\"fieldMetaOptions\\\":{\\\"isEnabled\\\":false,\\\"sigma\\\":3},\\\"type\\\":\\\"ORDINAL\\\"},\\\"type\\\":\\\"DYNAMIC\\\"},\\\"icon\\\":{\\\"options\\\":{\\\"value\\\":\\\"marker\\\"},\\\"type\\\":\\\"STATIC\\\"},\\\"iconOrientation\\\":{\\\"options\\\":{\\\"orientation\\\":0},\\\"type\\\":\\\"STATIC\\\"},\\\"iconSize\\\":{\\\"options\\\":{\\\"size\\\":6},\\\"type\\\":\\\"STATIC\\\"},\\\"labelBorderColor\\\":{\\\"options\\\":{\\\"color\\\":\\\"#FFFFFF\\\"},\\\"type\\\":\\\"STATIC\\\"},\\\"labelBorderSize\\\":{\\\"options\\\":{\\\"size\\\":\\\"SMALL\\\"}},\\\"labelColor\\\":{\\\"options\\\":{\\\"color\\\":\\\"#000000\\\"},\\\"type\\\":\\\"STATIC\\\"},\\\"labelSize\\\":{\\\"options\\\":{\\\"size\\\":14},\\\"type\\\":\\\"STATIC\\\"},\\\"labelText\\\":{\\\"options\\\":{\\\"value\\\":\\\"\\\"},\\\"type\\\":\\\"STATIC\\\"},\\\"lineColor\\\":{\\\"options\\\":{\\\"color\\\":\\\"#41937c\\\"},\\\"type\\\":\\\"STATIC\\\"},\\\"lineWidth\\\":{\\\"options\\\":{\\\"size\\\":1},\\\"type\\\":\\\"STATIC\\\"},\\\"symbolizeAs\\\":{\\\"options\\\":{\\\"value\\\":\\\"circle\\\"}}},\\\"type\\\":\\\"VECTOR\\\"},\\\"type\\\":\\\"VECTOR\\\",\\\"visible\\\":true}]\",\"mapStateJSON\":\"{\\\"zoom\\\":1.71,\\\"center\\\":{\\\"lon\\\":0,\\\"lat\\\":19.94277},\\\"timeFilters\\\":{\\\"from\\\":\\\"now-7d/d\\\",\\\"to\\\":\\\"now\\\"},\\\"refreshConfig\\\":{\\\"isPaused\\\":true,\\\"interval\\\":0},\\\"query\\\":{\\\"query\\\":\\\"\\\",\\\"language\\\":\\\"kuery\\\"},\\\"filters\\\":[],\\\"settings\\\":{\\\"autoFitToDataBounds\\\":false,\\\"backgroundColor\\\":\\\"#ffffff\\\",\\\"disableInteractive\\\":false,\\\"disableTooltipControl\\\":false,\\\"hideToolbarOverlay\\\":false,\\\"hideLayerControl\\\":false,\\\"hideViewControl\\\":false,\\\"initialLocation\\\":\\\"LAST_SAVED_LOCATION\\\",\\\"fixedLocation\\\":{\\\"lat\\\":0,\\\"lon\\\":0,\\\"zoom\\\":2},\\\"browserLocation\\\":{\\\"zoom\\\":2},\\\"maxZoom\\\":24,\\\"minZoom\\\":0,\\\"showScaleControl\\\":false,\\\"showSpatialFilters\\\":true,\\\"showTimesliderToggleButton\\\":true,\\\"spatialFiltersAlpa\\\":0.3,\\\"spatialFiltersFillColor\\\":\\\"#DA8B45\\\",\\\"spatialFiltersLineColor\\\":\\\"#DA8B45\\\"}}\",\"uiStateJSON\":\"{\\\"isLayerTOCOpen\\\":true,\\\"openTOCDetails\\\":[]}\",\"references\":[{\"name\":\"layer_1_join_0_index_pattern\",\"type\":\"index-pattern\",\"id\":\"90943e30-9a47-11e8-b64d-95841ca0b247\"}]}}}]", "timeRestore": false, "title": "by value maps", "version": 1 }, "coreMigrationVersion": "8.0.0", "id": "6bf77e10-e0e0-11eb-8303-39c9ad80bd90", "migrationVersion": { "dashboard": "7.14.0" }, "references": [ { "id": "90943e30-9a47-11e8-b64d-95841ca0b247", "name": "6939008d-992f-4f12-b3b0-487b499dd6d8:layer_1_source_index_pattern", "type": "index-pattern" }, { "id": "90943e30-9a47-11e8-b64d-95841ca0b247", "name": "87cd9c53-5e8b-4d56-89aa-47cd145e5297:layer_1_join_0_index_pattern", "type": "index-pattern" } ], "type": "dashboard", "updated_at": "2021-07-09T18:06:45.363Z", "version": "WzE2OTEsMV0=" } ```
elasticmachine commented 3 years ago

Pinging @elastic/kibana-gis (Team:Geo)

nreese commented 3 years ago

ping @pgayvallet

elasticmachine commented 3 years ago

Pinging @elastic/kibana-core (Team:Core)

pgayvallet commented 3 years ago

note: if you don't need all the context, you can jump to option 3 at the bottom)

We already have an issue about allowing type renaming (here: https://github.com/elastic/kibana/issues/91143).

However in the existing issue, the only need was to rename the type of ALL SOs of a given type to another in a given release, without further changes.

In theory, the SO migrator does already allow to change the type of a document during its migration. E.g during a 7.9.2 to 8.3.4 upgrade, foo could be renamed to bar during 8.2.1, and then continue with the bar migration starting at 8.2.2 up to 8.3.4. The most tricky (and currently unimplemented) part is to rename the references from other docs to the type that is going to be renamed. This is something that is already internally supported due to the ID rewrite used for the sharing SO feature (see SavedObjectType.convertToMultiNamespaceTypeVersion) via the reference type of migration, but it's not exposed to plugin yet. https://github.com/elastic/kibana/issues/91143#issuecomment-851994184 is a suggestion on how this could be provided as an API for type owners.

However the need here is more complex, due to the simple fact that we only want to migrate a subset of the visualization documents to map documents.

The SO DocumentMigrator's document migration is supposed to be working in isolation. The only input are the document you want to migrate, and its type metadata (where the migrations are registered for instance). For the ID rewrite, we already kinda were forced to cheat, as we're effectively using type metadata from other types to generate the migration map, to add migrations for reference rewrite for the version where other types were converted to multi-namespace.

Note that when thinking about the correct approach to potentially solve the issue at stake here, we also need to take SO import/export into account, as due to its amazing design, it's effectively possible to import objects without their references e.g

So we basically have to be able to handle both

I see two possible directions if we want to do that conversion, neither of them being perfect unfortunately.

Collect and expose a migration context

As we need to know which subset of visualizations will need to be converted to maps (for the references rewrite), we could (in addition to https://github.com/elastic/kibana/issues/91143), add the concept of 'migration context' that would be passed down to the migration functions, and add an API for type owners to register context gathering hooks.

We're already providing a SavedObjectMigrationContext object as the second parameter of any SavedObjectMigrationFn, which was more or less to future-proof against such needs. The harder part would be to define those context gathering hook APIs.

I think the easiest way would be to expose a handler that would have access to a (subset of the) SO client configured to hit the source index to let consumers perform any search/get queries they'll need to populate the context.

core.savedObjects.registerMigrationContextCollector = <T>(domain: string, collector: SavedObjectContextCollector<T>): void

type SavedObjectContextCollector<T> = (info: ContextCollectorInfo): Promise<T>;

type ContextCollectorInfo = {
  client: CollectorClient;
}

type CollectorClient = Pick<EsClient, 'whatever' | 'we' | 'want' | 'to' | 'expose'>

and then we'll update the SavedObjectMigrationContext type:

type SavedObjectMigrationContext {
  // already present
  readonly log: SavedObjectsMigrationLogger;
  readonly migrationVersion: string;
  readonly convertToMultiNamespaceTypeVersion?: string;
  // added
  readonly migrationContext: Record<string, unknown>; // I don't think we can easily do better than unknown
}

If that is achieved, in addition to allow type owners to register references type migrations somehow, I think we could do something.

The biggest problem here is the interactions with the import/export API: I'm not sure how we would be able to make that work with the import API, as we're using 'on-create' migration for this, meaning that the context will not have knowledge of the documents that were present in the import file.

E.g

Also note that as the documents are migrated atomically (we apply all migration for a given doc then move forward to the next one), the logic that determine which visualizations gonna need to be migrated to maps will have to work for any initial version (e.g from 7.1.0 to 8.0.0 as for 7.13.0 to 8.0.0), which I'm not sure would be acceptable?

cc @jportner in particular: I also see we're currently not calling the reference and convert type migrations during document-based migration, e.g import, and I feel it would like to change here? I don't even remember how that's supposed to work when importing from a version where a SO was not migrated to multi-NS to a version where it was?

Add a new post migration hook/API

A solution that could sounds simpler (probably due to the fact that's it is uglier) would be to expose a new API to register 'post-migration' hooks, where consumers would register a function to execute an arbitrary migration 'script' that would have access to the SO client.

The API could look like

core.savedObjects.registerPostMigration = (hook: SavedObjectPostMigrationHook): void;

type SavedObjectPostMigrationHook = (context: SavedObjectPostMigrationHookContext): Promise<SavedObjectPostMigrationHookResult>

interface SavedObjectPostMigrationHookContext {
  client: SavedObjectClientContract;
  // other things?
}

interface SavedObjectPostMigrationHookResult {
  // TBD
}

The pro is that is could easily be plugged into the import/export system, as we could just invoke the same hook after a import against the cluster with the imported objects.

The main con is that this would be performed after the whole migration, meaning that it would need to support manually applying the migration that may have been skipped.

To explain more in details:

In that case, the post hook will effectively only be called after the complete migration of all the documents up to v6. Meaning that the function that handles the conversion of vis to map will be called with all the vis already in v6, and will have to apply the map migration that were not applied.

Given that there is no reliable way to know which version the migration was initiated from (the migrationVersion of the vis that needs to be converted to maps will be v6), I'm not sure this approach is realistic.

EDIT : I need to stop thinking about SO migration during my week-ends, but:

Option 3: detect id renames automatically and perform reference rewrite in a second pass

Just though of it, but:

This goes in-between the two previous options.

Given that, in theory, the doc migration does support type renaming, I'm wondering if we couldn't just try to automatically detect id/type renaming during the migration, and trigger a references rename automatically as a second pass. I think it may actually be the easiest way.

The idea is the following:

`7.x.x`: (doc) => {
   if(shouldChangeToMap(doc)) {
      return {
         ...doc,
         attributes: convertToMap(doc.attributes),
         type: 'map',
         id: doc.id // I don't think we need to do anything in the id
      }
   }
}

This could even totally replace the reference type migration we're currently using I believe, as we would be doing it automatically? It would also address the problem about migrating docs with references to disabled types, as the migration rewrite is effectively performed when the OUTBOUND document is migrated.

One of the edge case for the migration would be if it fails between before the second-pass is applied, as in that case, if the docs are already in the target index, we would loose the information to allow references rewrite during the second attempt, so I think we need to do that in the temporary index to avoid such risk. It would invalidate https://github.com/elastic/kibana/issues/104080 btw.

cc @elastic/kibana-core @jportner @kobelb Please tell me if you see any other option, or if I said anything wrong (or forget anything)

@jportner especially, as our official honorary expert of the SO migration, what do you think about this third option? Did I miss anything that would make it not work?

joshdover commented 3 years ago

If we can make it work, I strongly prefer going with a solution that is handled directly by the Saved Object migration infrastructure over one that requires type owners to implement any custom interactions with Elasticsearch or SavedObjectsClient. I fear that opening that box would only increase the fragility of this system and make auditing our migration code for issues much harder. The more centralized, well-tested migration features we can provide at the Core level, the better IMO.

That said, I also want to be sure that we choose a path forward that will be coherent with any other migration features we need to add and support in the future.

With that in mind, of these options, I only think the 3rd one is going to be safe 'enough' to be considered. An open ended post-migration hook is likely the wrong long-term direction for the migration framework and IMO the migration context option seems unlikely to scale. Wouldn't you need to preserve the context for the entire migration, meaning that each batch of objects would make the context grow?

Also note that as the documents are migrated atomically (we apply all migration for a given doc then move forward to the next one), the logic that determine which visualizations gonna need to be migrated to maps will have to work for any initial version (e.g from 7.1.0 to 8.0.0 as for 7.13.0 to 8.0.0), which I'm not sure would be acceptable?

I'm curious if we should still be doing things this way. It does seem to me that if we have any migrations that have interactions across type boundaries that we're going to need to upgrade the documents version-by-version rather than type-by-type. This way the behavior is deterministic regardless of which version you're upgrading from.

Tangential to this topic, I'd like to have a dedicated API for anything like renaming types or splitting types over special behavior based on the return value. The purpose of a dedicated API would be to make these actions very explicitly chosen by the developer rather than something that is happening 'by accident' because of a return value. It also helps with discoverability of the features provided by the migration framework.

IMO an ideal "migration-on-rails" API might look something like this:

core.savedObjects.registerType({
  name: 'visualization',
  mappings: {},
  migrations: {
    '8.0.0': core.savedObjects
      .buildMigration()
      .renameField('fieldA', 'fieldB', 'default-value')
      .convertToType({
        newTypeName: 'map',
        filter: (doc) => isLegacyMap(doc),
        transform: (doc) => transformLegacyMapToNewMap(doc)
      })
     .customTransform((doc) => myCustomTransform(doc))
  }
});

The function that is returned by this buildMigration toolkit would need to be compatible with the existing interface, so we'd likely need to add some kind _tag field similar to fp-ts's Either type so that the migration framework can detect what state needs to be tracked for any follow up work (like updating references). Alternatively, we could replace the existing interface usages with buildMigration().customTransform() in one pass to make things simpler.

jportner commented 3 years ago

cc @jportner in particular: I also see we're currently not calling the reference and convert type migrations during document-based migration, e.g import, and I feel it would like to change here? I don't even remember how that's supposed to work when importing from a version where a SO was not migrated to multi-NS to a version where it was?

@pgayvallet Yeah that's intentional.

We only use the convert transforms (to change an object's ID, add origin ID, and add an alias) during an index migration. So it follows that we only use core-generated reference transforms during an index migration too.

If you have exported an object from 7.14, and that is converted in 8.0, and then you want to import it in 8.1: the import code will check for conflicts based on object ID and origin ID fields . This, coupled with the fact that imports do not include space information, means we don't have to attempt to convert-transform the document that we are importing. It will automatically be matched with an existing object with that origin ID and the user will be prompted to overwrite that.

This could even totally replace the reference type migration we're currently using I believe, as we would be doing it automatically?

I think so!* See (1) below

One of the edge case for the migration would be if it fails between before the second-pass is applied, as in that case, if the docs are already in the target index, we would loose the information to allow references rewrite during the second attempt, so I think we need to do that in the temporary index to avoid such risk.

Agree.

@jportner especially, as our official honorary expert of the SO migration, what do you think about this third option? Did I miss anything that would make it not work?

Two things come to mind:

1) "Changed Types/IDs" map must be cumulative and space-aware

For example, in a 7.14 instance, there are two visualizations and two dashboards (using raw document IDs for clarity):

{ type: 'visualization', id: '123' }
{ type: 'dashboard', id: '456', references: [{ type: 'visualization', id: '123' }] }
{ namespace: 'foo', type: 'visualization', id: '123' }
{ namespace: 'foo', type: 'dashboard', id: '456', references: [{ type: 'visualization', id: '123' }] }

These are single-namespace types. (Note: The plan is to convert them to "share-capable" namespaceType: 'multiple-isolated' in 8.0, but realistically we might not hit that goal)

The user upgrades their instance from 7.14 to 8.2. In 8.0 we convert the dashboards and visualizations, so their IDs change:

{ namespaces: ['default'], type: 'visualization', id: 'ffcee6e4' }
{ namespaces: ['default'], type: 'dashboard', id: '7d598c64', references: [{ type: 'visualization', id: '123' }] }
{ namespaces: ['foo'], type: 'visualization', id: 'dadccda1' }
{ namespaces: ['foo'], type: 'dashboard', id: '1fc8a162', references: [{ type: 'visualization', id: '123' }] }

In 8.2 we change the visualizations to maps:

{ namespaces: ['default'], type: 'map', id: 'ffcee6e4' }
{ namespaces: ['default'], type: 'dashboard', id: '7d598c64', references: [{ type: 'visualization', id: '123' }] }
{ namespaces: ['foo'], type: 'map', id: 'dadccda1' }
{ namespaces: ['foo'], type: 'dashboard', id: '1fc8a162', references: [{ type: 'visualization', id: '123' }] }

After all of that is complete, then in the second-pass we need to change any reference to visualization '123' in the default space to map 'ffcee6e4', and we need to change any reference to visualization '123' in the foo space to map 'dadccda1'. (We should also account for the "*" space.) Result:

{ namespaces: ['default'], type: 'map', id: 'ffcee6e4' }
{ namespaces: ['default'], type: 'dashboard', id: '7d598c64', references: [{ type: 'map', id: 'ffcee6e4' }] }
{ namespaces: ['foo'], type: 'map', id: 'dadccda1' }
{ namespaces: ['foo'], type: 'dashboard', id: '1fc8a162', references: [{ type: 'map', id: 'dadccda1' }] }

2) Consumers can no longer reliably write their own migrations that leverage reference types or IDs

This may be a minor point, but: the current algorithm for reference transforms has the advantage that it is always applied to target objects before other consumer-defined migrations for future versions.

If use this "second-pass", then consumers can run into problems when trying to manipulate references on their own. In the example above, if the consumer added a dashboard migration for 8.1 to, say, update the panelsJSON field based on the object's references -- that would use the outdated reference to visualization:123 instead of the new reference to map:ffcee6e4 and map:dadccda1.

In general I really don't like this fact -- the second-pass means that updating from 7.14 to 8.2 is not the same as upgrading from 7.14 to 8.0, then upgrading from 8.0 to 8.2.

We can't avoid this by doing a "second-pass" between each version, because all of the migrations are applied to the saved object on the client side before it is written to the temporary index.


With that in mind, of these options, I only think the 3rd one is going to be safe 'enough' to be considered. ... IMO an ideal "migration-on-rails" API might look something like this

@joshdover I agree

nreese commented 3 years ago

Another route would be to avoid the SO migration.

https://github.com/elastic/kibana/pull/105326 is a proof of concept of creating a tile_map visualization implementation that is just a thin wrapper around the MapEmbeddable. If we go this route, then the saved objects can stay as-is. Thoughts?

In this screen shot, the visualization is rendered using MapEmbeddable. There is no way to edit the visualization unless you open in maps.

Screen Shot 2021-07-12 at 2 28 43 PM

In this screen shot, the tile_map visualization saved object is rendered using MapEmbeddable.

Screen Shot 2021-07-12 at 2 39 53 PM
pgayvallet commented 3 years ago

@joshdover

It does seem to me that if we have any migrations that have interactions across type boundaries that we're going to need to upgrade the documents version-by-version rather than type-by-type

@jportner

Consumers can no longer reliably write their own migrations that leverage reference types or ID

Thanks joe for the detailed example. Following type/id rename in-between version doesn't seem that complicated. However I agree with you both that this is not ideal given the constraints it would create and the effective limitations that this will apply on the future migration APIs we would be able to expose to type owners.

Thinking it thought, it I had to choose one approach, I would still go with option 3. I believe, but you made it clear that even this one is not perfect. And I don't think going with a flawed design is something acceptable for the SO migration, unfortunately. There are just too much implications.

core.savedObjects
      .buildMigration()
      .renameField('fieldA', 'fieldB', 'default-value')
      .convertToType({
        newTypeName: 'map',
        filter: (doc) => isLegacyMap(doc),
        transform: (doc) => transformLegacyMapToNewMap(doc)
      })
     .customTransform((doc) => myCustomTransform(doc))

This would be some work, but could be a possibility (saying only could here because every time we want to touch the document migrator, we discover another limitation or problem that this would introduce)

Alternatively, we could replace the existing interface usages with buildMigration().customTransform()

If we do that, we would still have to limit what the current migration functions are allowed to do. Atm you can rename an id and/or a type from a custom migration function (even if effectively never used i believe/hope). If we were to add a fluent API and especially specific methods to changeId/renamedoc, I'm not sure we could still be able to support it from custom transforms, as those specific transformations would trigger additional actions from the migration system (e.g references handling when changing an id). I don't even understand why migration functions are allowed to migrate more than just attributes. It should have been the design since the beginning for the public API (but there was no such distinction in legacy)

Overall, the more I'm thinking about it, the more I think the migration should be handled on a per-version basis internally. When migrating to 7.8.1 to 7.9.2, we apply 7.9.0 to all docs, then 7.9.1, then 7.9.2. This would unlock a lot of possibility as it would make sure that migration from any version to any other one follow the exact same order/workflow, allowing a lot more of potential transformations, such as document splitting. I don't think I need to say that these would be massive changes to both the document migrator (no longer apply all migrations, but accepts a target version instead) AND the migration system (need a temporary index per version during migration, also the performance impact is significant as we multiply the overall duration by a factor equal to the number of migrations to apply).

The point is, even if we were to agree that this would be the correct approach (which I'm not even sure) designing and implementing it would realistically never be done for 8.0

I kinda hate it to be honest. id/type rename, document splitting, documents merging, all of these feels like valid migration needs to me, but our system simply can not handle them, and we don't have ANY correct solution to implement them (also, there is the import/export interactions that don't help a bit)

@nreese

In short, if you do have an alternative that doesn't rely on a migration for this conversion, I think it would probably be our best shot. I really don't know the actual implication of your adapter/wrapper pattern, but in a pure design point of view, it sounds good to me, and if you don't foresee any future problem this could cause later, I'd say go with it.

A question though: did you think about future map/vis migrations? How would you handle map type migrations if some are technically vis using an adapter?

joshdover commented 3 years ago

Sounds like we're generally in agreement that option 3 is likely the most viable path forward to supporting this feature before 8.0 given the current design of the system. In order to make a decision here we're going to need answers to:

joshdover commented 3 years ago

Overall, the more I'm thinking about it, the more I think the migration should be handled on a per-version basis internally. When migrating to 7.8.1 to 7.9.2, we apply 7.9.0 to all docs, then 7.9.1, then 7.9.2. This would unlock a lot of possibility as it would make sure that migration from any version to any other one follow the exact same order/workflow, allowing a lot more of potential transformations, such as document splitting. I don't think I need to say that these would be massive changes to both the document migrator (no longer apply all migrations, but accepts a target version instead) AND the migration system (need a temporary index per version during migration, also the performance impact is significant as we multiply the overall duration by a factor equal to the number of migrations to apply).

The point is, even if we were to agree that this would be the correct approach (which I'm not even sure) designing and implementing it would realistically never be done for 8.0

I kinda hate it to be honest. id/type rename, document splitting, documents merging, all of these feels like valid migration needs to me, but our system simply can not handle them, and we don't have ANY correct solution to implement them (also, there is the import/export interactions that don't help a bit)

I agree 100%. We need to be performing migrations in the most deterministic way possible so that we can be sure that they continue working regardless of the upgrade path a cluster takes. As it is, we already have too many variables (eg. disabled plugins) that can impact what operations will happen during a migration.

I think this change will need to be a focus for the team during 8.x so that we can support these "table stakes" migration features. While performance is a concern, I think if we can make upgrades rock-solid and seamless, there are other potential ways to work around a slower migration process (eg. auto-scheduled upgrades during off-peak hours).

nreese commented 3 years ago

In short, if you do have an alternative that doesn't rely on a migration for this conversion, I think it would probably be our best shot. I really don't know the actual implication of your adapter/wrapper pattern, but in a pure design point of view, it sounds good to me, and if you don't foresee any future problem this could cause later, I'd say go with it.

I will keep on pushing it to make sure there are no unseen implementation problems.

did you think about future map/vis migrations? How would you handle map type migrations if some are technically vis using an adapter?

The tile_map visualization saved object is converted to maps configuration in code when run. We will add functional tests make sure changes to either visualization saved objects or maps configurations do not break things. Then we can just update the logic that converts tile_map visualization saved object is converted to maps configuration if needed.

nreese commented 3 years ago

How painful would @nreese's workaround be for users? IMO, this seems like a pretty confusing UX unless we could make it more automatic. For instance, is it possible to automatically redirect the user to the Maps application with the converted legacy map when they try editing a tile_map or region_map?

I think the UI concerns are limited. For users consuming tile_map and region_map visualizations in Dashboard and Canvas, there will be no UI problems as the panel will just render as a normal map embeddable.

I agree there may be some confusion when users edit tile_map and region_map visualizations, but, again I think this is limited. Since 7.10, tile_map and region_map have displayed a large ugly deprecation message encouraging them to migrate to Maps.

@kmartastic What are your thoughts on UI issues with the proposed solution of making tile_map visualizations wrappers around map embeddable?

nreese commented 3 years ago

@pgayvallet @joshdover @jportner

I wanted to give an update on moving tile_map and region_map visualization implementations to MapEmbeddable. The PR has progressed well and it has been verified that the implementation will work for both tile_map and region_map. The only UI wrinkle is that editing tile_map and region_map visualizations will require users to manually save the tile_map and region_map visualization as a Map saved object and then manually replace the tile_map and region_map panel in Dashboard or canvas with the new Map saved object. @elastic/kibana-gis discussed this at todays team sync and is ok moving forward with the solution proposed in https://github.com/elastic/kibana/pull/105326.

This means, that for the 8.0 time frame, from @elastic/kibana-gis's perspective, saved object migration work needed to support migrating tile_map and region_map visualization saved objects to maps saved objects can be pushed down in priority.

elasticmachine commented 1 year ago

Pinging @elastic/kibana-presentation (Team:Presentation)

pgayvallet commented 2 months ago

With serverless / ZDT, we now know for sure that type renames are technically not doable. Given that and the fact that 8.0 is way far behind us now, I'll assume we no longer need this and will close it.