Plugin installation was unsuccessful due to error (v6.7.1)

[pavdmyt]

I'm trying to install plugin for kibana v6.7.1 and having troubles after Optimizing and caching browser bundles... step. Running kibana in K8s, using official docker image: docker.elastic.co/kibana/kibana:6.7.1. Here is steps to reproduce the issue:

$ kubectl exec -ti <kibana-pod> -- sh
sh-4.2$
sh-4.2$ bin/kibana-plugin list

sh-4.2$
sh-4.2$
sh-4.2$ bin/kibana-plugin install https://github.com/pjhampton/kibana-prometheus-exporter/releases/download/6.7.1/kibana-prometheus-exporter-6.7.1.zip
Attempting to transfer from https://github.com/pjhampton/kibana-prometheus-exporter/releases/download/6.7.1/kibana-prometheus-exporter-6.7.1.zip
Transferring 1131929 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...
Plugin installation was unsuccessful due to error "Command failed: /usr/share/kibana/node/bin/node /usr/share/kibana/src/cli --env.name=production --optimize.useBundleCache=false --server.autoListen=false --plugins.initialize=false
events.js:174
      throw er; // Unhandled 'error' event
      ^

Error: read ECONNRESET
    at Pipe.onStreamRead (internal/stream_base_commons.js:111:27)
Emitted 'error' event at:
    at emitErrorNT (internal/streams/destroy.js:82:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)
    at process._tickCallback (internal/process/next_tick.js:63:19)

{"type":"log","@timestamp":"2020-05-29T16:02:29Z","tags":["info","plugins-service"],"pid":471,"message":"Plugin initialization disabled."}
{"type":"log","@timestamp":"2020-05-29T16:02:58Z","tags":["warning","elasticsearch","config","deprecation"],"pid":471,"message":"Config key \"url\" is deprecated. It has been replaced with \"hosts\""}
{"type":"log","@timestamp":"2020-05-29T16:02:59Z","tags":["info","optimize"],"pid":471,"message":"Optimizing and caching bundles for graph, monitoring, space_selector, login, logout, logged_out, ml, dashboardViewer, apm, maps, canvas, infra, uptime, kibana-prometheus-exporter, kibana, stateSessionStorageRedirect, status_page and timelion. This may take a few minutes"}
"
sh-4.2$

[pjhampton]

Thanks for reporting @pavdmyt I will look into this tonight

[pjhampton]

Hey @pavdmyt

I couldn't replicate:

It builds just fine and I looked at the download stats. There are downloads so this suggests to me that people are running this version in prod ok. Could it be have been a timeout or something to do with your flags?

--optimize.useBundleCache=false --server.autoListen=false --plugins.initialize=false

Please try again and make sure it wasn't just an unfortunate event.

[pavdmyt]

@pjhampton I've tried multiple times with different tweaks and no success in our K8s setup.

I thought that issue is somehow related to using official kibana docker image: docker.elastic.co/kibana/kibana:6.7.1. Tried to reproduce it using small 2-node cluster runnning via docker-compose:

version: '2.2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.7.1
    container_name: elasticsearch
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet

  elasticsearch2:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.7.1
    container_name: elasticsearch2
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.zen.ping.unicast.hosts=elasticsearch"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata2:/usr/share/elasticsearch/data
    networks:
      - esnet

  kib01:
    image: docker.elastic.co/kibana/kibana:6.7.1
    container_name: kibana
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://elasticsearch:9200
      ELASTICSEARCH_HOSTS: http://elasticsearch:9200
    networks:
      - esnet

volumes:
  esdata1:
    driver: local
  esdata2:
    driver: local

networks:
  esnet:

After running docker-compose up I'm having successfully starting ES nodes and kibana instance:

$ docker ps                                                                                                
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                              NAMES
e6b28a2167d3        docker.elastic.co/elasticsearch/elasticsearch:6.7.1   "/usr/local/bin/dock…"   58 seconds ago      Up 57 seconds       9200/tcp, 9300/tcp                 elasticsearch2
9f40de0e747f        docker.elastic.co/elasticsearch/elasticsearch:6.7.1   "/usr/local/bin/dock…"   58 seconds ago      Up 57 seconds       0.0.0.0:9200->9200/tcp, 9300/tcp   elasticsearch
589d684cdb43        docker.elastic.co/kibana/kibana:6.7.1                 "/usr/local/bin/kiba…"   58 seconds ago      Up 57 seconds       0.0.0.0:5601->5601/tcp             kibana

Trying to reproduce plugin installation failure:

$ docker exec -ti kibana bash                                   
bash-4.2$
bash-4.2$
bash-4.2$ bin/kibana-plugin list

bash-4.2$
bash-4.2$ bin/kibana-plugin install https://github.com/pjhampton/kibana-prometheus-exporter/releases/download/6.7.1/kibana-prometheus-exporter-6.7.1.zip
Attempting to transfer from https://github.com/pjhampton/kibana-prometheus-exporter/releases/download/6.7.1/kibana-prometheus-exporter-6.7.1.zip
Transferring 1131929 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...
Plugin installation complete

It seems that issue is caused not by the image docker.elastic.co/kibana/kibana:6.7.1 itself, but K8s setup for kibana. We're using stable/kibana v2.3.1 helm chart and probably some misconfiguration hides there for plugin installation.

I believe we can close this issue. Thx!

[pjhampton]

That's really interesting @pavdmyt - thanks for getting into a deep investigation and writing back. If you do find the root cause can you please come back and post it here in the future? I'm sure there are people who would be very grateful for your knowledge on running Kibana on k8s.

Thanks again ✨