search

elastic / ecctl

Apache License 2.0
57 stars 53 forks source link

Flag --generate-update-payload throws segmentation violation when showing a deployment #413

Closed liza-mae closed 3 years ago

liza-mae commented 3 years ago

./ecctl version

Version:               v1.1.0
Client API Version:    2.8.0-ms48
Go version:            go1.15.2
Git commit:            a2293c25
Built:                 Thu 03 Dec 23:21:49 2020
OS/Arch:               linux / amd64

./ecctl deployment show (id) --generate-update-payload

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xc3a571]

goroutine 1 [running]:
github.com/elastic/cloud-sdk-go/pkg/api/deploymentapi.parseElasticsearchGetResponse(0xc00051e540, 0xc0002e2080, 0xc000420140, 0x0)
    /Users/karen/go/pkg/mod/github.com/elastic/cloud-sdk-go@v1.1.1-0.20201201215440-1aa8a65e9add/pkg/api/deploymentapi/update_payload.go:100 +0xf1
github.com/elastic/cloud-sdk-go/pkg/api/deploymentapi.NewUpdateRequest(0xc000420180, 0x7fff71459769)
    /Users/karen/go/pkg/mod/github.com/elastic/cloud-sdk-go@v1.1.1-0.20201201215440-1aa8a65e9add/pkg/api/deploymentapi/update_payload.go:39 +0x19b
github.com/elastic/ecctl/cmd/deployment.glob..func8(0x1702c00, 0xc000366540, 0x1, 0x2, 0x0, 0x0)
    /Users/karen/go/src/github.com/elastic/ecctl/cmd/deployment/show.go:95 +0x512
github.com/spf13/cobra.(*Command).execute(0x1702c00, 0xc000366520, 0x2, 0x2, 0x1702c00, 0xc000366520)
    /Users/karen/go/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:850 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0x17011c0, 0xf8e80d, 0xc00021fe80, 0xc0c0fa)
    /Users/karen/go/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:958 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
    /Users/karen/go/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:895
github.com/elastic/ecctl/cmd.Execute(0x10a005c, 0x6, 0xf91342, 0xa, 0x10d5e00, 0x28, 0x10bf1d0, 0x18, 0x10a0054, 0x5, ...)
    /Users/karen/go/src/github.com/elastic/ecctl/cmd/root.go:96 +0xa6
main.main()
    /Users/karen/go/src/github.com/elastic/ecctl/main.go:39 +0x105
liza-mae commented 3 years ago

This is blocking me.

liza-mae commented 3 years ago

cc: @karencfv @marclop

liza-mae commented 3 years ago

I went back all the way to 1.0.0 version, but it still fails (also notice above it is referencing /users/karen path while this one references /home/runner path) 

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xc35a5e]

goroutine 1 [running]:
github.com/elastic/cloud-sdk-go/pkg/api/deploymentapi.parseElasticsearchGetResponse(0xc0001fe580, 0xc000414800, 0x0, 0x0)
    /home/runner/go/pkg/mod/github.com/elastic/cloud-sdk-go@v1.0.0/pkg/api/deploymentapi/update_payload.go:94 +0xde
github.com/elastic/cloud-sdk-go/pkg/api/deploymentapi.NewUpdateRequest(0xc00032a1b0, 0x7ffc28927767)
    /home/runner/go/pkg/mod/github.com/elastic/cloud-sdk-go@v1.0.0/pkg/api/deploymentapi/update_payload.go:39 +0x19c
github.com/elastic/ecctl/cmd/deployment.glob..func8(0x187be20, 0xc000450ba0, 0x1, 0x2, 0x0, 0x0)
    /home/runner/work/ecctl/ecctl/cmd/deployment/show.go:95 +0x527
github.com/spf13/cobra.(*Command).execute(0x187be20, 0xc000450b60, 0x2, 0x2, 0x187be20, 0xc000450b60)
    /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842 +0x460
github.com/spf13/cobra.(*Command).ExecuteC(0x1879420, 0xd, 0xc00016fe58, 0xc046eb)
    /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x349
github.com/spf13/cobra.(*Command).Execute(...)
    /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
github.com/elastic/ecctl/cmd.Execute(0x10fec7c, 0x6, 0xfc85ff, 0x5, 0x1138de0, 0x28, 0x1121960, 0x18, 0x10fec6c, 0x5, ...)
    /home/runner/work/ecctl/ecctl/cmd/root.go:96 +0xc0
main.main()
    /home/runner/work/ecctl/ecctl/main.go:39 +0xfe
karencfv commented 3 years ago

Hi @liza-mae 👋 I have tried the flag with several deployments with different versions and I have not been able to reproduce this bug. Would this deployment you're trying to update be very old by any chance?

In the mean time, the --generate-update-payload is not the only way to generate an update JSON definition. If you run ecctl deployment update --help there are instructions you can follow to creata a JSON definition. There is also an example in our API's documentation https://www.elastic.co/guide/en/cloud/current/Deployment_-_CRUD.html#ec_request_example_7

karencfv commented 3 years ago

Could you also paste here what you get from running ecctl deployment show <deployment id>?

liza-mae commented 3 years ago

@karencfv thanks - yes let me get that info for you, it should not be an old deployment but let me verify that and get back to you.

liza-mae commented 3 years ago

I am running 7.10.1 BC 2 I think, so it is not that old using ECE 2.7.0. Let me know if this needs to be updated. Below is the show deployment id output and it works for other flags I tried just not --generate-update-payload not sure why. 

./ecctl deployment show 294076fca23e9bf29e03a311eb295bbd
{
  "healthy": true,
  "id": "294076fca23e9bf29e03a311eb295bbd",
  "metadata": {
    "hidden": false,
    "last_modified": "2020-12-04T19:48:55.205Z",
    "last_resource_plan_modified": "0001-01-01T00:00:00.000Z",
    "owner_id": "admin",
    "system_owned": false
  },
  "name": "ESTF__fe336bf4-b9de-42a5-a430-5f8ca9fa69bb",
  "resources": {
    "apm": [],
    "appsearch": [],
    "elasticsearch": [
      {
        "id": "466f2731afc2421390ae55af9ed9ac8a",
        "info": {
          "associated_apm_clusters": [],
          "associated_appsearch_clusters": [],
          "associated_enterprise_search_clusters": [],
          "associated_kibana_clusters": [
            {
              "enabled": true,
              "kibana_id": "02ac804ec47c43cfb0fe6725d54ea41f"
            }
          ],
          "cluster_id": "466f2731afc2421390ae55af9ed9ac8a",
          "cluster_name": "ESTF Elasticsearch Cluster",
          "deployment_id": "294076fca23e9bf29e03a311eb295bbd",
          "elasticsearch": {
            "blocking_issues": {
              "cluster_level": [],
              "healthy": true,
              "index_level": []
            },
            "healthy": true,
            "master_info": {
              "healthy": true,
              "instances_with_no_master": [],
              "masters": [
                {
                  "instances": [
                    "instance-0000000000",
                    "instance-0000000001"
                  ],
                  "master_instance_name": "instance-0000000000",
                  "master_node_id": "VUSPaOLXSPmM8lJlBEszYw"
                }
              ]
            },
            "shard_info": {
              "available_shards": [
                {
                  "instance_name": "instance-0000000000",
                  "shard_count": 8
                },
                {
                  "instance_name": "instance-0000000001",
                  "shard_count": 8
                }
              ],
              "healthy": true,
              "unavailable_replicas": [
                {
                  "instance_name": "instance-0000000000",
                  "replica_count": 0
                },
                {
                  "instance_name": "instance-0000000001",
                  "replica_count": 0
                }
              ],
              "unavailable_shards": [
                {
                  "instance_name": "instance-0000000000",
                  "shard_count": 0
                },
                {
                  "instance_name": "instance-0000000001",
                  "shard_count": 0
                }
              ]
            }
          },
          "external_links": [
            {
              "id": "cluster-logs",
              "label": "Elasticsearch Logs",
              "uri": "https://18512dd951ab4ad08d349f09aeb22fd7.10.128.0.76.ip.es.io:9243/app/kibana#/discover?_g=(time:(from:now-1h,mode:quick,to:now))\u0026_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'cluster-logs-*',key:ece.cluster,negate:!f,params:(query:'466f2731afc2421390ae55af9ed9ac8a',type:phrase),type:phrase,value:'466f2731afc2421390ae55af9ed9ac8a'),query:(match:(ece.cluster:(query:'466f2731afc2421390ae55af9ed9ac8a',type:phrase))))),index:'cluster-logs-*',interval:auto,query:(language:lucene,query:''),sort:!('@timestamp',desc))"
            },
            {
              "id": "metricbeat",
              "label": "Elasticsearch Metrics",
              "uri": "https://18512dd951ab4ad08d349f09aeb22fd7.10.128.0.76.ip.es.io:9243/app/kibana#/dashboard/AV4REOpp5NkDleZmzKkE?_a=(filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'metricbeat-*',key:ece.cluster,negate:!f,params:(query:'466f2731afc2421390ae55af9ed9ac8a',type:phrase),type:phrase,value:'466f2731afc2421390ae55af9ed9ac8a'),query:(match:(ece.cluster:(query:'466f2731afc2421390ae55af9ed9ac8a',type:phrase))))))\u0026_g=(time:(from:now-1h,mode:quick,to:now))"
            },
            {
              "id": "proxy-logs",
              "label": "Proxy logs",
              "uri": "https://18512dd951ab4ad08d349f09aeb22fd7.10.128.0.76.ip.es.io:9243/app/kibana#/discover?_g=(time:(from:now-1h,mode:quick,to:now))\u0026_a=(columns:!(status_code,request_method,request_path,request_length,response_length,response_time),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'proxy-logs-*',key:handling_cluster,negate:!f,params:(query:'466f2731afc2421390ae55af9ed9ac8a',type:phrase),type:phrase,value:'466f2731afc2421390ae55af9ed9ac8a'),query:(match:(handling_cluster:(query:'466f2731afc2421390ae55af9ed9ac8a',type:phrase))))),index:'proxy-logs-*',interval:auto,query:(language:lucene,query:''),sort:!('@timestamp',desc))"
            }
          ],
          "healthy": true,
          "metadata": {
            "cloud_id": "ESTF_Elasticsearch_Cluster:MTAuMTI4LjAuNzYuaXAuZXMuaW86OTI0MyQ0NjZmMjczMWFmYzI0MjEzOTBhZTU1YWY5ZWQ5YWM4YSQwMmFjODA0ZWM0N2M0M2NmYjBmZTY3MjVkNTRlYTQxZg==",
            "endpoint": "466f2731afc2421390ae55af9ed9ac8a.10.128.0.76.ip.es.io",
            "last_modified": "2020-12-04T19:48:55.205Z",
            "ports": {
              "http": 9200,
              "https": 9243,
              "transport_passthrough": 9400
            },
            "version": 8
          },
          "plan_info": {
            "healthy": true,
            "history": []
          },
          "snapshots": {
            "count": 0,
            "healthy": true,
            "latest_end_time": "0001-01-01T00:00:00.000Z",
            "latest_successful_end_time": "0001-01-01T00:00:00.000Z",
            "recent_success": false,
            "scheduled_time": "0001-01-01T00:00:00.000Z"
          },
          "status": "started",
          "system_alerts": [],
          "topology": {
            "healthy": true,
            "instances": [
              {
                "allocator_id": "10.128.0.76",
                "container_started": true,
                "disk": {
                  "disk_space_available": 198310,
                  "disk_space_used": 55575,
                  "storage_multiplier": 32
                },
                "healthy": true,
                "instance_configuration": {
                  "id": "data.default",
                  "name": "data.default",
                  "resource": "memory"
                },
                "instance_name": "instance-0000000000",
                "maintenance_mode": false,
                "memory": {
                  "instance_capacity": 1024,
                  "memory_pressure": 25,
                  "native_memory_pressure": 75
                },
                "service_id": "VUSPaOLXSPmM8lJlBEszYw",
                "service_roles": [
                  "data",
                  "master",
                  "ingest"
                ],
                "service_running": true,
                "service_version": "7.10.1-928e89a5",
                "zone": "ece-zone-1"
              },
              {
                "allocator_id": "10.128.0.76",
                "container_started": true,
                "disk": {
                  "disk_space_available": 198310,
                  "disk_space_used": 55575,
                  "storage_multiplier": 4
                },
                "healthy": true,
                "instance_configuration": {
                  "id": "ml",
                  "name": "ml",
                  "resource": "memory"
                },
                "instance_name": "instance-0000000001",
                "maintenance_mode": false,
                "memory": {
                  "instance_capacity": 1024,
                  "memory_pressure": 19,
                  "native_memory_pressure": 66
                },
                "service_id": "ggMlpRPVSyGY-Q0lK6MhHQ",
                "service_roles": [
                  "ml"
                ],
                "service_running": true,
                "service_version": "7.10.1-928e89a5",
                "zone": "ece-zone-1"
              }
            ]
          }
        },
        "ref_id": "estf-es-cluster",
        "region": "ece-region"
      }
    ],
    "enterprise_search": [],
    "kibana": [
      {
        "elasticsearch_cluster_ref_id": "estf-es-cluster",
        "id": "02ac804ec47c43cfb0fe6725d54ea41f",
        "info": {
          "cluster_id": "02ac804ec47c43cfb0fe6725d54ea41f",
          "cluster_name": "ESTF Kibana Instance",
          "deployment_id": "294076fca23e9bf29e03a311eb295bbd",
          "elasticsearch_cluster": {
            "elasticsearch_id": "466f2731afc2421390ae55af9ed9ac8a"
          },
          "external_links": [
            {
              "id": "cluster-logs",
              "label": "Elasticsearch Logs",
              "uri": "https://18512dd951ab4ad08d349f09aeb22fd7.10.128.0.76.ip.es.io:9243/app/kibana#/discover?_g=(time:(from:now-1h,mode:quick,to:now))\u0026_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'cluster-logs-*',key:ece.cluster,negate:!f,params:(query:'02ac804ec47c43cfb0fe6725d54ea41f',type:phrase),type:phrase,value:'02ac804ec47c43cfb0fe6725d54ea41f'),query:(match:(ece.cluster:(query:'02ac804ec47c43cfb0fe6725d54ea41f',type:phrase))))),index:'cluster-logs-*',interval:auto,query:(language:lucene,query:''),sort:!('@timestamp',desc))"
            },
            {
              "id": "metricbeat",
              "label": "Elasticsearch Metrics",
              "uri": "https://18512dd951ab4ad08d349f09aeb22fd7.10.128.0.76.ip.es.io:9243/app/kibana#/dashboard/AV4REOpp5NkDleZmzKkE?_a=(filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'metricbeat-*',key:ece.cluster,negate:!f,params:(query:'02ac804ec47c43cfb0fe6725d54ea41f',type:phrase),type:phrase,value:'02ac804ec47c43cfb0fe6725d54ea41f'),query:(match:(ece.cluster:(query:'02ac804ec47c43cfb0fe6725d54ea41f',type:phrase))))))\u0026_g=(time:(from:now-1h,mode:quick,to:now))"
            },
            {
              "id": "proxy-logs",
              "label": "Proxy logs",
              "uri": "https://18512dd951ab4ad08d349f09aeb22fd7.10.128.0.76.ip.es.io:9243/app/kibana#/discover?_g=(time:(from:now-1h,mode:quick,to:now))\u0026_a=(columns:!(status_code,request_method,request_path,request_length,response_length,response_time),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'proxy-logs-*',key:handling_cluster,negate:!f,params:(query:'02ac804ec47c43cfb0fe6725d54ea41f',type:phrase),type:phrase,value:'02ac804ec47c43cfb0fe6725d54ea41f'),query:(match:(handling_cluster:(query:'02ac804ec47c43cfb0fe6725d54ea41f',type:phrase))))),index:'proxy-logs-*',interval:auto,query:(language:lucene,query:''),sort:!('@timestamp',desc))"
            }
          ],
          "healthy": true,
          "metadata": {
            "endpoint": "02ac804ec47c43cfb0fe6725d54ea41f.10.128.0.76.ip.es.io",
            "last_modified": "2020-12-04T19:48:50.631Z",
            "ports": {
              "http": 9200,
              "https": 9243,
              "transport_passthrough": 9400
            },
            "version": 7
          },
          "plan_info": {
            "healthy": true,
            "history": []
          },
          "status": "started",
          "topology": {
            "healthy": true,
            "instances": [
              {
                "allocator_id": "10.128.0.76",
                "container_started": true,
                "healthy": true,
                "instance_configuration": {
                  "id": "kibana",
                  "name": "kibana",
                  "resource": "memory"
                },
                "instance_name": "instance-0000000000",
                "maintenance_mode": false,
                "memory": {
                  "instance_capacity": 1024,
                  "native_memory_pressure": 22
                },
                "service_roles": null,
                "service_running": true,
                "service_version": "7.10.1-928e89a5",
                "zone": "ece-zone-1"
              }
            ]
          }
        },
        "ref_id": "estf-kibana-instance",
        "region": "ece-region"
      }
    ]
  }
}
karencfv commented 3 years ago

Thanks @liza-mae ! Sorry, I gave you the wrong command it should have been ecctl deployment show <deployment id> --plans

By the way, did you delete your deployment? I tried looking for it in production and staging but it wasn't there.

karencfv commented 3 years ago

Oh I just noticed it says "ece-region" 😄 I take it this is your own ce-aws environment?

liza-mae commented 3 years ago

@karencfv yes it is my own environment, below is the output. 

./ecctl deployment show 294076fca23e9bf29e03a311eb295bbd --plans
{
  "healthy": true,
  "id": "294076fca23e9bf29e03a311eb295bbd",
  "metadata": {
    "hidden": false,
    "last_modified": "2020-12-04T19:48:55.205Z",
    "last_resource_plan_modified": "2020-12-04T19:49:16.735Z",
    "owner_id": "admin",
    "system_owned": false
  },
  "name": "ESTF__fe336bf4-b9de-42a5-a430-5f8ca9fa69bb",
  "resources": {
    "apm": [],
    "appsearch": [],
    "elasticsearch": [
      {
        "id": "466f2731afc2421390ae55af9ed9ac8a",
        "info": {
          "associated_apm_clusters": [],
          "associated_appsearch_clusters": [],
          "associated_enterprise_search_clusters": [],
          "associated_kibana_clusters": [
            {
              "enabled": true,
              "kibana_id": "02ac804ec47c43cfb0fe6725d54ea41f"
            }
          ],
          "cluster_id": "466f2731afc2421390ae55af9ed9ac8a",
          "cluster_name": "ESTF Elasticsearch Cluster",
          "deployment_id": "294076fca23e9bf29e03a311eb295bbd",
          "elasticsearch": {
            "blocking_issues": {
              "cluster_level": [],
              "healthy": true,
              "index_level": []
            },
            "healthy": true,
            "master_info": {
              "healthy": true,
              "instances_with_no_master": [],
              "masters": [
                {
                  "instances": [
                    "instance-0000000000",
                    "instance-0000000001"
                  ],
                  "master_instance_name": "instance-0000000000",
                  "master_node_id": "VUSPaOLXSPmM8lJlBEszYw"
                }
              ]
            },
            "shard_info": {
              "available_shards": [
                {
                  "instance_name": "instance-0000000000",
                  "shard_count": 8
                },
                {
                  "instance_name": "instance-0000000001",
                  "shard_count": 8
                }
              ],
              "healthy": true,
              "unavailable_replicas": [
                {
                  "instance_name": "instance-0000000000",
                  "replica_count": 0
                },
                {
                  "instance_name": "instance-0000000001",
                  "replica_count": 0
                }
              ],
              "unavailable_shards": [
                {
                  "instance_name": "instance-0000000000",
                  "shard_count": 0
                },
                {
                  "instance_name": "instance-0000000001",
                  "shard_count": 0
                }
              ]
            }
          },
          "external_links": [
            {
              "id": "cluster-logs",
              "label": "Elasticsearch Logs",
              "uri": "https://18512dd951ab4ad08d349f09aeb22fd7.10.128.0.76.ip.es.io:9243/app/kibana#/discover?_g=(time:(from:now-1h,mode:quick,to:now))\u0026_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'cluster-logs-*',key:ece.cluster,negate:!f,params:(query:'466f2731afc2421390ae55af9ed9ac8a',type:phrase),type:phrase,value:'466f2731afc2421390ae55af9ed9ac8a'),query:(match:(ece.cluster:(query:'466f2731afc2421390ae55af9ed9ac8a',type:phrase))))),index:'cluster-logs-*',interval:auto,query:(language:lucene,query:''),sort:!('@timestamp',desc))"
            },
            {
              "id": "metricbeat",
              "label": "Elasticsearch Metrics",
              "uri": "https://18512dd951ab4ad08d349f09aeb22fd7.10.128.0.76.ip.es.io:9243/app/kibana#/dashboard/AV4REOpp5NkDleZmzKkE?_a=(filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'metricbeat-*',key:ece.cluster,negate:!f,params:(query:'466f2731afc2421390ae55af9ed9ac8a',type:phrase),type:phrase,value:'466f2731afc2421390ae55af9ed9ac8a'),query:(match:(ece.cluster:(query:'466f2731afc2421390ae55af9ed9ac8a',type:phrase))))))\u0026_g=(time:(from:now-1h,mode:quick,to:now))"
            },
            {
              "id": "proxy-logs",
              "label": "Proxy logs",
              "uri": "https://18512dd951ab4ad08d349f09aeb22fd7.10.128.0.76.ip.es.io:9243/app/kibana#/discover?_g=(time:(from:now-1h,mode:quick,to:now))\u0026_a=(columns:!(status_code,request_method,request_path,request_length,response_length,response_time),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'proxy-logs-*',key:handling_cluster,negate:!f,params:(query:'466f2731afc2421390ae55af9ed9ac8a',type:phrase),type:phrase,value:'466f2731afc2421390ae55af9ed9ac8a'),query:(match:(handling_cluster:(query:'466f2731afc2421390ae55af9ed9ac8a',type:phrase))))),index:'proxy-logs-*',interval:auto,query:(language:lucene,query:''),sort:!('@timestamp',desc))"
            }
          ],
          "healthy": true,
          "metadata": {
            "cloud_id": "ESTF_Elasticsearch_Cluster:MTAuMTI4LjAuNzYuaXAuZXMuaW86OTI0MyQ0NjZmMjczMWFmYzI0MjEzOTBhZTU1YWY5ZWQ5YWM4YSQwMmFjODA0ZWM0N2M0M2NmYjBmZTY3MjVkNTRlYTQxZg==",
            "endpoint": "466f2731afc2421390ae55af9ed9ac8a.10.128.0.76.ip.es.io",
            "last_modified": "2020-12-04T19:48:55.205Z",
            "ports": {
              "http": 9200,
              "https": 9243,
              "transport_passthrough": 9400
            },
            "version": 8
          },
          "plan_info": {
            "current": {
              "attempt_end_time": "2020-12-04T19:48:55.211Z",
              "attempt_start_time": "2020-12-04T19:48:11.925Z",
              "healthy": true,
              "plan": {
                "cluster_topology": [
                  {
                    "instance_configuration_id": "data.default",
                    "memory_per_node": 1024,
                    "node_count_per_zone": 1,
                    "node_type": {
                      "data": true,
                      "ingest": true,
                      "master": true,
                      "ml": false
                    },
                    "zone_count": 1
                  },
                  {
                    "elasticsearch": {
                      "system_settings": {
                        "auto_create_index": true,
                        "destructive_requires_name": false,
                        "enable_close_index": true,
                        "monitoring_collection_interval": -1,
                        "monitoring_history_duration": "3d",
                        "reindex_whitelist": [],
                        "scripting": {
                          "inline": {
                            "enabled": true
                          },
                          "stored": {
                            "enabled": true
                          }
                        },
                        "use_disk_threshold": true
                      }
                    },
                    "instance_configuration_id": "ml",
                    "node_type": {
                      "data": false,
                      "ingest": false,
                      "master": false,
                      "ml": true
                    },
                    "size": {
                      "resource": "memory",
                      "value": 1024
                    },
                    "zone_count": 1
                  },
                  {
                    "elasticsearch": {
                      "system_settings": {
                        "auto_create_index": true,
                        "destructive_requires_name": false,
                        "enable_close_index": true,
                        "monitoring_collection_interval": -1,
                        "monitoring_history_duration": "3d",
                        "reindex_whitelist": [],
                        "scripting": {
                          "inline": {
                            "enabled": true
                          },
                          "stored": {
                            "enabled": true
                          }
                        },
                        "use_disk_threshold": true
                      }
                    },
                    "instance_configuration_id": "coordinating",
                    "node_type": {
                      "data": false,
                      "ingest": true,
                      "master": false
                    },
                    "size": {
                      "resource": "memory",
                      "value": 0
                    },
                    "zone_count": 1
                  },
                  {
                    "elasticsearch": {
                      "system_settings": {
                        "auto_create_index": true,
                        "destructive_requires_name": false,
                        "enable_close_index": true,
                        "monitoring_collection_interval": -1,
                        "monitoring_history_duration": "3d",
                        "reindex_whitelist": [],
                        "scripting": {
                          "inline": {
                            "enabled": true
                          },
                          "stored": {
                            "enabled": true
                          }
                        },
                        "use_disk_threshold": true
                      }
                    },
                    "instance_configuration_id": "master",
                    "node_type": {
                      "data": false,
                      "ingest": false,
                      "master": true
                    },
                    "size": {
                      "resource": "memory",
                      "value": 0
                    },
                    "zone_count": 1
                  }
                ],
                "deployment_template": {
                  "id": "default"
                },
                "elasticsearch": {
                  "version": "7.10.1-928e89a5"
                }
              },
              "plan_attempt_id": "3879e79a-3211-4830-b468-c317babd7407",
              "plan_attempt_log": [],
              "plan_end_time": "0001-01-01T00:00:00.000Z",
              "source": {
                "action": "deployments.create-deployment",
                "admin_id": "admin",
                "date": "2020-12-04T19:48:11.369Z",
                "facilitator": "adminconsole",
                "remote_addresses": [
                  "10.128.0.76"
                ]
              }
            },
            "healthy": true,
            "history": []
          },
          "snapshots": {
            "count": 0,
            "healthy": true,
            "latest_end_time": "0001-01-01T00:00:00.000Z",
            "latest_successful_end_time": "0001-01-01T00:00:00.000Z",
            "recent_success": false,
            "scheduled_time": "0001-01-01T00:00:00.000Z"
          },
          "status": "started",
          "system_alerts": [],
          "topology": {
            "healthy": true,
            "instances": [
              {
                "allocator_id": "10.128.0.76",
                "container_started": true,
                "disk": {
                  "disk_space_available": 198310,
                  "disk_space_used": 56810,
                  "storage_multiplier": 32
                },
                "healthy": true,
                "instance_configuration": {
                  "id": "data.default",
                  "name": "data.default",
                  "resource": "memory"
                },
                "instance_name": "instance-0000000000",
                "maintenance_mode": false,
                "memory": {
                  "instance_capacity": 1024,
                  "memory_pressure": 26,
                  "native_memory_pressure": 75
                },
                "service_id": "VUSPaOLXSPmM8lJlBEszYw",
                "service_roles": [
                  "data",
                  "master",
                  "ingest"
                ],
                "service_running": true,
                "service_version": "7.10.1-928e89a5",
                "zone": "ece-zone-1"
              },
              {
                "allocator_id": "10.128.0.76",
                "container_started": true,
                "disk": {
                  "disk_space_available": 198310,
                  "disk_space_used": 56810,
                  "storage_multiplier": 4
                },
                "healthy": true,
                "instance_configuration": {
                  "id": "ml",
                  "name": "ml",
                  "resource": "memory"
                },
                "instance_name": "instance-0000000001",
                "maintenance_mode": false,
                "memory": {
                  "instance_capacity": 1024,
                  "memory_pressure": 19,
                  "native_memory_pressure": 66
                },
                "service_id": "ggMlpRPVSyGY-Q0lK6MhHQ",
                "service_roles": [
                  "ml"
                ],
                "service_running": true,
                "service_version": "7.10.1-928e89a5",
                "zone": "ece-zone-1"
              }
            ]
          }
        },
        "ref_id": "estf-es-cluster",
        "region": "ece-region"
      }
    ],
    "enterprise_search": [],
    "kibana": [
      {
        "elasticsearch_cluster_ref_id": "estf-es-cluster",
        "id": "02ac804ec47c43cfb0fe6725d54ea41f",
        "info": {
          "cluster_id": "02ac804ec47c43cfb0fe6725d54ea41f",
          "cluster_name": "ESTF Kibana Instance",
          "deployment_id": "294076fca23e9bf29e03a311eb295bbd",
          "elasticsearch_cluster": {
            "elasticsearch_id": "466f2731afc2421390ae55af9ed9ac8a"
          },
          "external_links": [
            {
              "id": "cluster-logs",
              "label": "Elasticsearch Logs",
              "uri": "https://18512dd951ab4ad08d349f09aeb22fd7.10.128.0.76.ip.es.io:9243/app/kibana#/discover?_g=(time:(from:now-1h,mode:quick,to:now))\u0026_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'cluster-logs-*',key:ece.cluster,negate:!f,params:(query:'02ac804ec47c43cfb0fe6725d54ea41f',type:phrase),type:phrase,value:'02ac804ec47c43cfb0fe6725d54ea41f'),query:(match:(ece.cluster:(query:'02ac804ec47c43cfb0fe6725d54ea41f',type:phrase))))),index:'cluster-logs-*',interval:auto,query:(language:lucene,query:''),sort:!('@timestamp',desc))"
            },
            {
              "id": "metricbeat",
              "label": "Elasticsearch Metrics",
              "uri": "https://18512dd951ab4ad08d349f09aeb22fd7.10.128.0.76.ip.es.io:9243/app/kibana#/dashboard/AV4REOpp5NkDleZmzKkE?_a=(filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'metricbeat-*',key:ece.cluster,negate:!f,params:(query:'02ac804ec47c43cfb0fe6725d54ea41f',type:phrase),type:phrase,value:'02ac804ec47c43cfb0fe6725d54ea41f'),query:(match:(ece.cluster:(query:'02ac804ec47c43cfb0fe6725d54ea41f',type:phrase))))))\u0026_g=(time:(from:now-1h,mode:quick,to:now))"
            },
            {
              "id": "proxy-logs",
              "label": "Proxy logs",
              "uri": "https://18512dd951ab4ad08d349f09aeb22fd7.10.128.0.76.ip.es.io:9243/app/kibana#/discover?_g=(time:(from:now-1h,mode:quick,to:now))\u0026_a=(columns:!(status_code,request_method,request_path,request_length,response_length,response_time),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'proxy-logs-*',key:handling_cluster,negate:!f,params:(query:'02ac804ec47c43cfb0fe6725d54ea41f',type:phrase),type:phrase,value:'02ac804ec47c43cfb0fe6725d54ea41f'),query:(match:(handling_cluster:(query:'02ac804ec47c43cfb0fe6725d54ea41f',type:phrase))))),index:'proxy-logs-*',interval:auto,query:(language:lucene,query:''),sort:!('@timestamp',desc))"
            }
          ],
          "healthy": true,
          "metadata": {
            "endpoint": "02ac804ec47c43cfb0fe6725d54ea41f.10.128.0.76.ip.es.io",
            "last_modified": "2020-12-04T19:48:50.631Z",
            "ports": {
              "http": 9200,
              "https": 9243,
              "transport_passthrough": 9400
            },
            "version": 7
          },
          "plan_info": {
            "current": {
              "attempt_end_time": "2020-12-04T19:49:16.735Z",
              "attempt_start_time": "2020-12-04T19:48:12.562Z",
              "healthy": true,
              "plan": {
                "cluster_topology": [
                  {
                    "instance_configuration_id": "kibana",
                    "memory_per_node": 1024,
                    "node_count_per_zone": 1,
                    "zone_count": 1
                  }
                ],
                "kibana": {
                  "system_settings": {
                    "elasticsearch_password": "CcP6Dh91xEHwmFh8brOA3D_U",
                    "elasticsearch_url": "http://466f2731afc2421390ae55af9ed9ac8a.containerhost:9244",
                    "elasticsearch_username": "found-internal-kibana4-server"
                  },
                  "user_settings_override_json": {
                    "data.search.aggs.shardDelay.enabled": true,
                    "home.disableWelcomeScreen": true,
                    "logging.verbose": true,
                    "map.proxyElasticMapsServiceInMaps": true,
                    "server.uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
                    "telemetry.banner": false,
                    "vis_type_timelion.ui.enabled": true,
                    "xpack.encryptedSavedObjects.encryptionKey": "DkdXazszSCYexXqz4YktBGHCRkV6hyNK",
                    "xpack.maps.preserveDrawingBuffer": true,
                    "xpack.maps.showMapsInspectorAdapter": true,
                    "xpack.reporting.capture.maxAttempts": 1,
                    "xpack.reporting.capture.networkPolicy.rules": [
                      {
                        "allow": true,
                        "protocol": "http:"
                      },
                      {
                        "allow": false,
                        "host": "via.placeholder.com"
                      },
                      {
                        "allow": true,
                        "protocol": "https:"
                      },
                      {
                        "allow": true,
                        "protocol": "data:"
                      },
                      {
                        "allow": false
                      }
                    ],
                    "xpack.reporting.capture.timeouts.openUrl": 60000,
                    "xpack.reporting.csv.checkForFormulas": false,
                    "xpack.reporting.csv.enablePanelActionDownload": true,
                    "xpack.reporting.csv.maxSizeBytes": 2850,
                    "xpack.reporting.queue.pollInterval": 3000,
                    "xpack.security.encryptionKey": "wuGNaIhoMpk5sO4UBxgr3NyW1sFcLgIf",
                    "xpack.security.session.idleTimeout": 3600000
                  },
                  "version": "7.10.1-928e89a5"
                },
                "transient": {
                  "strategy": {
                    "autodetect": {}
                  }
                }
              },
              "plan_attempt_id": "c20a8039-7b99-459b-8b02-47a7580b868a",
              "plan_attempt_log": [],
              "plan_end_time": "0001-01-01T00:00:00.000Z",
              "source": {
                "action": "deployments.create-deployment",
                "admin_id": "admin",
                "date": "2020-12-04T19:48:11.369Z",
                "facilitator": "adminconsole",
                "remote_addresses": [
                  "10.128.0.76"
                ]
              }
            },
            "healthy": true,
            "history": []
          },
          "status": "started",
          "topology": {
            "healthy": true,
            "instances": [
              {
                "allocator_id": "10.128.0.76",
                "container_started": true,
                "healthy": true,
                "instance_configuration": {
                  "id": "kibana",
                  "name": "kibana",
                  "resource": "memory"
                },
                "instance_name": "instance-0000000000",
                "maintenance_mode": false,
                "memory": {
                  "instance_capacity": 1024,
                  "native_memory_pressure": 22
                },
                "service_roles": null,
                "service_running": true,
                "service_version": "7.10.1-928e89a5",
                "zone": "ece-zone-1"
              }
            ]
          }
        },
        "ref_id": "estf-kibana-instance",
        "region": "ece-region"
      }
    ]
  }
}
karencfv commented 3 years ago

Thank you! I see what happened, for some reason one of your Elasticsearch cluster topology elements is missing size as you can see here:

{
  "instance_configuration_id": "data.default",
  "memory_per_node": 1024,
  "node_count_per_zone": 1,
  "node_type": {
    "data": true,
    "ingest": true,
    "master": true,
    "ml": false
  },
  "zone_count": 1
},

# This should have the following before "zone_count":
"size": {
  "resource": "memory",
  "value": 0
},

I'm not really sure how this happened as they're all supposed to have the "value" set to 0 instead of omitted like here, but 🤷‍♀️. I'll write up a patch to handle this case anyway. Thanks for helping me out debug this!

marclop commented 3 years ago

strangely the plan has memory_per_node set instead of size.value. If that's something created through the API, the payload should be changed to use the Size object Karen mentions.