Permission errors in packages after server optimizing

[jbudz]

Kibana version: 5.0, 4.6

Original install method (e.g. download page, yum, from source, etc.): yum, deb

Description of the problem including expected versus actual behavior: Plugin installs with deb and rpm packages typically result in bundles owned by root and file permissions set to 644. This is sufficient to run, but in certain situations such as disabling a plugin, the kibana server will attempt to regenerate these bundles. This results in permission denied errors because the server is not ran as root and can't write to these existing bundles.

Steps to reproduce:

1. Install Kibana using the .deb or .rpm package as root user on Linux
2. Install x-pack also as root user
3. In config/kibana.yml, set xpack.monitoring.enabled: false
4. Restart server

Provide logs and/or server output (if relevant): Oct 24 21:55:58 kbdeb64 kibana[24273]: {"type":"log","@timestamp":"2016-10-24T21:55:58Z","tags":["fatal"],"pid":24273,"level":"fatal","message":"EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/graph.entry.js'","error":{"message":"EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/graph.entry.js'","name":"Error","stack":"Error: EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/graph.entry.js'\n at Error (native)","code":"EACCES"}}

Solution:

If installing the .deb or .rpm package (includes if you use apt-get, dpkg, rpm, yum) follow updated instructions in https://www.elastic.co/guide/en/kibana/6.2/installing-xpack-kb.html Specifically the part that says to install x-pack as the local kibana user that is created by installing Kibana sudo -u kibana bin/kibana-plugin install x-pack

[jbudz]

This isn't a straightforward fix, there are a few options:

Long term:

• https://github.com/elastic/kibana/issues/7322, ideal solution

Short term:

• Put the optimization step for plugin installs behind a flag (or https://github.com/elastic/kibana/issues/6057). When the server first starts, it will optimize then and files will have proper permissions
  • I'd like to remove this step altogether, but it can be useful to re-bundle after plugin install to share with other machines
  • Another benefit to this is we can install multiple plugins without re-bundling in between each
• Document installing plugins as user kibana: sudo -u kibana bin/kibana-plugin install x-pack
• Post plugin install chown - this would be prone to breaking.

Anything else I'm missing? I think documentation with https://github.com/elastic/kibana/issues/8845 makes sense now, and then putting the optimization step behind a flag for 6.0 (or 5.1)?, and then long term #7322.

/cc @epixa @tylersmalley

[epixa]

I agree that we should do the docs changes right now, but I think we should aggressively pursue #7322 now rather than doing an interim optimization flag change.

[jlecour]

Hi,

I usually use a read-only filesystem for /usr (as it should not contain variable data outside of packages installed by the package manager.

With Kibana, after install, I can start the server since nothing can be written to /usr/share/kibana/optimize. The solution is to put this "optimize" directory in a path designed to be variable. I've moved it to /var/lib/kibana/optimize and now it works properly.

It should not be a breaking change since this data is only an internal implementation detail, only of Kibana's concern.

Do you think it should be the new way of doing this ?

[spinus]

I saw kibana src that it's using DATA_PATH env variable, maybe would be ok to move optimized data there or to TEMP?

[jbudz]

@spinus it's an interesting idea, I do think there's something there with moving the optimize files to a different folder for organizations sake. It won't fix this issue with the reproduction steps above though because the plugin installer ends up creating root owned files that the kibana server can't read. A different folder would still have that issue -there's nothing chowning the data folder post install.

[ghost]

chown -R kibana:kibana /opt/kibana/optimize/bundles chmod g+s /opt/kibana/optimize/bundles

Fixes this issue for me. chmod g+s will ensure that even files created by root will have kibana group membership which allows elk to start. This gets me past the error. tags":["info","optimize"],"pid":330,"message":"Optimizing and caching bundles for graph, monitoring, ml, kibana, stateSessionStorageRedirect, timelion and status_page. This may take a few minutes"} Looks like it gets stuck on optimizing.

There are files without permissions created ... ???

-rw-rw-r-- 1 kibana kibana 1.1K Jul 26 20:50 a3b2c394347b7dba49ca15464bb9e234.svg -rw-rw-r-- 1 kibana kibana 1.2K Jul 26 20:50 ccd9193f3b8af43575b17f6b00df04ae.svg -rw-rw-r-- 1 kibana kibana 1.7K Jul 26 20:50 d1c95c8e85228e93547df753065c0867.svg -rw-rw-r-- 1 kibana kibana 45K Jul 26 20:50 e18bbf611f2a2e43afc071aa2f4e1512.ttf -rw-rw-r-- 1 kibana kibana 1.4K Jul 26 20:50 f1db327547974915c7a903d2e5e9eedc.svg -rw-rw-r-- 1 kibana kibana 20K Jul 26 20:50 f4769f9bdb7466be65088239c12046d1.eot -rw-rw-r-- 1 kibana kibana 23K Jul 26 20:50 fa2772327f55d8198301fdb8bcfc8158.woff -rw-rw-r-- 1 kibana kibana 68K Jul 26 20:50 45c73723862c6fc5eb3d6961db2d71fb.eot -rw-rw-r-- 1 kibana kibana 63K Jul 26 20:50 4b5a84aaf1c9485e060c503a0ff8cadb.woff2 -rw-rw-r-- 1 kibana kibana 3.0K Jul 26 20:50 4eae70b44c4e3b69f2357bff8f1e67f1.svg -rw-rw-r-- 1 kibana kibana 1.6K Jul 26 20:50 6199ab7146b1eb859db9ffd2b9b786d3.svg -rw-rw-r-- 1 kibana kibana 3.2K Jul 26 20:50 9ad9ca7ab44828ee57b75fdd76a84567.svg -rw-rw-r-- 1 kibana kibana 1.4K Jul 26 20:50 a04c09b6eda01f84b583ed59e958afdc.svg -rw-rw-r-- 1 kibana kibana 80K Jul 26 20:50 dfb02f8f6d0cedc009ee5887cc68f1f3.woff -rw-r--r-- 1 kibana kibana 1.8K Jul 26 20:45 logout.entry.js -rw-r--r-- 1 kibana kibana 1.8K Jul 26 20:45 login.entry.js drwxrwxr-x 1 kibana kibana 15 Jul 26 20:43 src drwxrwxr-x 1 kibana kibana 43 Jul 26 20:43 .. ?????????? ? ? ? ? ? kibana.bundle.js ?????????? ? ? ? ? ? stateSessionStorageRedirect.bundle.js ?????????? ? ? ? ? ? status_page.bundle.js ?????????? ? ? ? ? ? timelion.bundle.js

[hatdropper1977]

@jbudz I installed X-Pack Kibana as root and got the permission error on graph.entry.js. I then removed the x-pack plugin and re-installed using:

sudo -u kibana bin/kibana-plugin install x-pack

I then get the following error:

Plugin installation was unsuccessful due to error "EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/graph.entry.js'"

[jbudz]

@hatdropper1977 it sounds like it could be leftover from the previous install. Can you try clearing out the optimize/bundles?

mv optimize/bundles optimize/old_bundles
sudo -u kibana bin/kibana-plugin install x-pack

[hatdropper1977]

@jbudz I deleted the /usr/share/kibana directory, re-installed the RPM, tried again and that worked. Thanks!

[LeeDr]

I'm seeing this now on 6.2.0-SNAPSHOT (from 6.x branch) on CentOS with the rpm package installed. I'm installing x-pack like this;

==> centos_rpm: ++ /usr/share/kibana/bin/kibana-plugin install file:////vagrant/qa/x-pack-6.2.0-SNAPSHOT.zip
==> centos_rpm: Attempting to transfer from file:////vagrant/qa/x-pack-6.2.0-SNAPSHOT.zip
==> centos_rpm: Transferring 320091529 bytes
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm: .
==> centos_rpm:
==> centos_rpm: Transfer complete
==> centos_rpm: Retrieving metadata from plugin archive
==> centos_rpm: Extracting plugin archive
==> centos_rpm: Extraction complete
==> centos_rpm: Optimizing and caching browser bundles...
==> centos_rpm: Plugin installation complete
==> centos_rpm:
==> centos_rpm: real    13m11.129s

My kibana.yml contains this;

[root@localhost qa]# grep -v "#" /etc/kibana/kibana.yml
elasticsearch.url: https://elastic:changeit@localhost:9200
logging.verbose: true
server.ssl.enabled: true
elasticsearch.username: kibana
elasticsearch.password: changeit
xpack.reporting.encryptionKey: ThisIsReportingEncryptionKey1234
xpack.security.encryptionKey: ThisIsSecurityEncryptionKey12345
xpack.reporting.queue.timeout: 60000
xpack.reporting.capture.browser.type: chromium
xpack.reporting.capture.browser.chromium.disableSandbox: true
server.host: 0.0.0.0
server.ssl.certificate: /etc/kibana/kibana.crt
server.ssl.key: /etc/kibana/kibana.key
elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/ca.crt"]

And my log file has this;

[root@localhost qa]# cat /var/log/kibana/kibana.stdout
{"type":"log","@timestamp":"2017-12-27T17:04:42Z","tags":["plugin","debug"],"pid":13583,"path":"/usr/share/kibana/plugins/x-pack","message":"Found plugin at /usr/share/kibana/plugins/x-pack"}
{"type":"log","@timestamp":"2017-12-27T17:04:42Z","tags":["plugin","debug"],"pid":13583,"path":"/usr/share/kibana/src/core_plugins/console","message":"Found plugin at /usr/share/kibana/src/core_plugins/console"}
{"type":"log","@timestamp":"2017-12-27T17:04:42Z","tags":["plugin","debug"],"pid":13583,"path":"/usr/share/kibana/src/core_plugins/elasticsearch","message":"Found plugin at /usr/share/kibana/src/core_plugins/elasticsearch"}
{"type":"log","@timestamp":"2017-12-27T17:04:42Z","tags":["plugin","debug"],"pid":13583,"path":"/usr/share/kibana/src/core_plugins/input_control_vis","message":"Found plugin at /usr/share/kibana/src/core_plugins/input_control_vis"}
{"type":"log","@timestamp":"2017-12-27T17:04:42Z","tags":["plugin","debug"],"pid":13583,"path":"/usr/share/kibana/src/core_plugins/kbn_doc_views","message":"Found plugin at /usr/share/kibana/src/core_plugins/kbn_doc_views"}
{"type":"log","@timestamp":"2017-12-27T17:04:42Z","tags":["plugin","debug"],"pid":13583,"path":"/usr/share/kibana/src/core_plugins/kbn_vislib_vis_types","message":"Found plugin at /usr/share/kibana/src/core_plugins/kbn_vislib_vis_types"}
{"type":"log","@timestamp":"2017-12-27T17:04:42Z","tags":["plugin","debug"],"pid":13583,"path":"/usr/share/kibana/src/core_plugins/kibana","message":"Found plugin at /usr/share/kibana/src/core_plugins/kibana"}
{"type":"log","@timestamp":"2017-12-27T17:04:42Z","tags":["plugin","debug"],"pid":13583,"path":"/usr/share/kibana/src/core_plugins/markdown_vis","message":"Found plugin at /usr/share/kibana/src/core_plugins/markdown_vis"}
{"type":"log","@timestamp":"2017-12-27T17:04:42Z","tags":["plugin","debug"],"pid":13583,"path":"/usr/share/kibana/src/core_plugins/metric_vis","message":"Found plugin at /usr/share/kibana/src/core_plugins/metric_vis"}
{"type":"log","@timestamp":"2017-12-27T17:04:42Z","tags":["plugin","debug"],"pid":13583,"path":"/usr/share/kibana/src/core_plugins/metrics","message":"Found plugin at /usr/share/kibana/src/core_plugins/metrics"}
{"type":"log","@timestamp":"2017-12-27T17:04:42Z","tags":["plugin","debug"],"pid":13583,"path":"/usr/share/kibana/src/core_plugins/region_map","message":"Found plugin at /usr/share/kibana/src/core_plugins/region_map"}
{"type":"log","@timestamp":"2017-12-27T17:04:42Z","tags":["plugin","debug"],"pid":13583,"path":"/usr/share/kibana/src/core_plugins/spy_modes","message":"Found plugin at /usr/share/kibana/src/core_plugins/spy_modes"}
{"type":"log","@timestamp":"2017-12-27T17:04:42Z","tags":["plugin","debug"],"pid":13583,"path":"/usr/share/kibana/src/core_plugins/state_session_storage_redirect","message":"Found plugin at /usr/share/kibana/src/core_plugins/state_session_storage_redirect"}
{"type":"log","@timestamp":"2017-12-27T17:04:42Z","tags":["plugin","debug"],"pid":13583,"path":"/usr/share/kibana/src/core_plugins/status_page","message":"Found plugin at /usr/share/kibana/src/core_plugins/status_page"}
{"type":"log","@timestamp":"2017-12-27T17:04:42Z","tags":["plugin","debug"],"pid":13583,"path":"/usr/share/kibana/src/core_plugins/table_vis","message":"Found plugin at /usr/share/kibana/src/core_plugins/table_vis"}
{"type":"log","@timestamp":"2017-12-27T17:04:42Z","tags":["plugin","debug"],"pid":13583,"path":"/usr/share/kibana/src/core_plugins/tagcloud","message":"Found plugin at /usr/share/kibana/src/core_plugins/tagcloud"}
{"type":"log","@timestamp":"2017-12-27T17:04:42Z","tags":["plugin","debug"],"pid":13583,"path":"/usr/share/kibana/src/core_plugins/tile_map","message":"Found plugin at /usr/share/kibana/src/core_plugins/tile_map"}
{"type":"log","@timestamp":"2017-12-27T17:04:42Z","tags":["plugin","debug"],"pid":13583,"path":"/usr/share/kibana/src/core_plugins/timelion","message":"Found plugin at /usr/share/kibana/src/core_plugins/timelion"}
{"type":"error","@timestamp":"2017-12-27T17:04:43Z","tags":["fatal"],"pid":13583,"level":"fatal","error":{"message":"EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/ml.entry.js'","name":"Error","stack":"Error: EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/ml.
entry.js'\n    at Error (native)","code":"EACCES"},"message":"EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/ml.entry.js'"}

And my kibana/optimize directory (sorted) looks like this;

[root@localhost qa]# ls -al /usr/share/kibana/optimize/bundles/ | sort
drwxr-xr-x 3 kibana kibana       16 Dec 27 16:49 src
drwxr-xr-x 3 kibana kibana     4096 Dec 27 17:03 .
drwxr-xr-x 4 kibana kibana       59 Dec 27 16:49 ..
-rw-r--r-- 1 root   root   10289801 Dec 27 17:03 kibana.bundle.js
-rw-r--r-- 1 root   root       1196 Dec 27 17:03 logout.bundle.js
-rw-r--r-- 1 root   root       1366 Dec 27 17:03 ad8bf7c645b85923d0ffa8e0a5a926e0.svg
-rw-r--r-- 1 root   root       1370 Dec 27 17:03 2d849bdfebcdd45cd620cb3dde289007.svg
-rw-r--r-- 1 root   root       1566 Dec 27 17:03 744e93515eb75dfd98add1274a85e938.svg
-rw-r--r-- 1 root   root       1802 Dec 27 16:51 apm.entry.js
-rw-r--r-- 1 root   root       1815 Dec 27 16:51 login.entry.js
-rw-r--r-- 1 root   root       1816 Dec 27 16:51 logout.entry.js
-rw-r--r-- 1 root   root       1816 Dec 27 16:51 monitoring.entry.js
-rw-r--r-- 1 root   root       1982 Dec 27 17:03 0db73ed585dad9085a1258af1acd11b4.svg
-rw-r--r-- 1 root   root       2089 Dec 27 16:51 ml.entry.js
-rw-r--r-- 1 root   root       2092 Dec 27 16:51 graph.entry.js
-rw-r--r-- 1 root   root       2351 Dec 27 17:03 5e19b5d332782c61ef0aa61ae55fe839.svg
-rw-r--r-- 1 root   root     294579 Dec 27 17:03 stateSessionStorageRedirect.bundle.js
-rw-r--r-- 1 root   root     298170 Dec 27 17:03 login.bundle.js
-rw-r--r-- 1 root   root     310999 Dec 27 17:03 status_page.bundle.js
-rw-r--r-- 1 root   root       3350 Dec 27 16:51 dashboardViewer.entry.js
-rw-r--r-- 1 root   root     338798 Dec 27 17:03 stateSessionStorageRedirect.style.css
-rw-r--r-- 1 root   root     341004 Dec 27 17:03 login.style.css
-rw-r--r-- 1 root   root     358401 Dec 27 17:03 graph.style.css
-rw-r--r-- 1 root   root     362146 Dec 27 17:03 apm.style.css
-rw-r--r-- 1 root   root       3703 Dec 27 17:03 58d1156276e82458dae9756614e96899.svg
-rw-r--r-- 1 root   root     373759 Dec 27 17:03 monitoring.style.css
-rw-r--r-- 1 root   root    4495296 Dec 27 17:03 graph.bundle.js
-rw-r--r-- 1 root   root    4778509 Dec 27 17:03 timelion.bundle.js
-rw-r--r-- 1 root   root     486036 Dec 27 17:03 ml.style.css
-rw-r--r-- 1 root   root     508656 Dec 27 17:03 dashboardViewer.style.css
-rw-r--r-- 1 root   root    5270439 Dec 27 17:03 ml.bundle.js
-rw-r--r-- 1 root   root    6454054 Dec 27 17:03 monitoring.bundle.js
-rw-r--r-- 1 root   root    7716883 Dec 27 17:03 apm.bundle.js
-rw-r--r-- 1 root   root         77 Dec 27 17:03 logout.style.css
-rw-r--r-- 1 root   root    8214433 Dec 27 17:03 dashboardViewer.bundle.js
-rw-rw-r-- 1 kibana kibana   108738 Dec 27 17:03 89889688147bd7575d6327160d64e760.svg
-rw-rw-r-- 1 kibana kibana     1089 Dec 27 17:03 a3b2c394347b7dba49ca15464bb9e234.svg
-rw-rw-r-- 1 kibana kibana     1150 Dec 27 17:03 ccd9193f3b8af43575b17f6b00df04ae.svg
-rw-rw-r-- 1 kibana kibana     1173 Dec 27 17:03 05997d91a7ba91b3eb8774591478f045.svg
-rw-rw-r-- 1 kibana kibana     1271 Dec 27 17:03 4be5b8808b5216ee6b6a1a4a29493e32.svg
-rw-rw-r-- 1 kibana kibana     1301 Dec 27 17:03 283abe380441c6725a8faf00adfd138c.svg
-rw-rw-r-- 1 kibana kibana     1368 Dec 27 17:03 f1db327547974915c7a903d2e5e9eedc.svg
-rw-rw-r-- 1 kibana kibana     1380 Dec 27 17:03 5a2ae67159a441fd4b6cf30c0c6cc5b1.svg
-rw-rw-r-- 1 kibana kibana   138204 Dec 27 17:03 7c87870ab40d63cfb8870c1f183f9939.ttf
-rw-rw-r-- 1 kibana kibana     1409 Dec 27 17:03 a04c09b6eda01f84b583ed59e958afdc.svg
-rw-rw-r-- 1 kibana kibana     1587 Dec 27 17:03 3211ece03a463a51c356a406827c30c0.svg
-rw-rw-r-- 1 kibana kibana     1624 Dec 27 17:03 6199ab7146b1eb859db9ffd2b9b786d3.svg
-rw-rw-r-- 1 kibana kibana     1626 Dec 27 17:03 86581dc57044730966dee64f92133ee2.svg
-rw-rw-r-- 1 kibana kibana  1662878 Dec 27 17:03 commons.bundle.js
-rw-rw-r-- 1 kibana kibana     1698 Dec 27 17:03 97bdb00c6bb0f0858161ef433ea62640.svg
-rw-rw-r-- 1 kibana kibana     1718 Dec 27 17:03 d1c95c8e85228e93547df753065c0867.svg
-rw-rw-r-- 1 kibana kibana     1747 Dec 27 17:03 74725229b8c17a39c35d6cf26a989557.svg
-rw-rw-r-- 1 kibana kibana    18028 Dec 27 17:03 448c34a56d699c29117adc64c43affeb.woff2
-rw-rw-r-- 1 kibana kibana     1807 Dec 27 17:03 69d89e51f62b6a582c311c35c0f778aa.svg
-rw-rw-r-- 1 kibana kibana     1818 Dec 27 16:51 status_page.entry.js
-rw-rw-r-- 1 kibana kibana     1825 Dec 27 16:51 stateSessionStorageRedirect.entry.js
-rw-rw-r-- 1 kibana kibana    20127 Dec 27 17:03 f4769f9bdb7466be65088239c12046d1.eot
-rw-rw-r-- 1 kibana kibana     2095 Dec 27 16:51 timelion.entry.js
-rw-rw-r-- 1 kibana kibana    23424 Dec 27 17:03 fa2772327f55d8198301fdb8bcfc8158.woff
-rw-rw-r-- 1 kibana kibana     2597 Dec 27 17:03 71676fac241b6e6e51614074cf72152c.svg
-rw-rw-r-- 1 kibana kibana     2995 Dec 27 17:03 4eae70b44c4e3b69f2357bff8f1e67f1.svg
-rw-rw-r-- 1 kibana kibana     3057 Dec 27 17:03 ae11252ad19209059498cac1cd1addd7.svg
-rw-rw-r-- 1 kibana kibana     3181 Dec 27 17:03 9ad9ca7ab44828ee57b75fdd76a84567.svg
-rw-rw-r-- 1 kibana kibana   355981 Dec 27 17:03 76a4f23c6be74fd309e0d0fd2c27a5de.svg
-rw-rw-r-- 1 kibana kibana     3727 Dec 27 17:03 0c6dccf6e8d60a35330c60a8831b1c08.svg
-rw-rw-r-- 1 kibana kibana   374716 Dec 27 17:03 status_page.style.css
-rw-rw-r-- 1 kibana kibana   376614 Dec 27 17:03 timelion.style.css
-rw-rw-r-- 1 kibana kibana     4447 Dec 27 16:51 kibana.entry.js
-rw-rw-r-- 1 kibana kibana     4457 Dec 27 17:03 3c9862715546cd524edf3b9fa7516994.svg
-rw-rw-r-- 1 kibana kibana    45404 Dec 27 17:03 e18bbf611f2a2e43afc071aa2f4e1512.ttf
-rw-rw-r-- 1 kibana kibana      457 Dec 27 17:03 428687c29242d0d2f3d7ac7e09942921.svg
-rw-rw-r-- 1 kibana kibana    47064 Dec 27 17:03 08dd79443c24d83e569939cc78508c5e.js
-rw-rw-r-- 1 kibana kibana      569 Dec 27 17:03 31c86fc56ac59def71b7d7cff315270b.svg
-rw-rw-r-- 1 kibana kibana   602724 Dec 27 17:03 kibana.style.css
-rw-rw-r-- 1 kibana kibana    64464 Dec 27 17:03 4b5a84aaf1c9485e060c503a0ff8cadb.woff2
-rw-rw-r-- 1 kibana kibana    68875 Dec 27 17:03 45c73723862c6fc5eb3d6961db2d71fb.eot
-rw-rw-r-- 1 kibana kibana      750 Dec 27 17:03 1f5de7e3ff25f0b0acccfa328794c7ea.svg
-rw-rw-r-- 1 kibana kibana    81284 Dec 27 17:03 dfb02f8f6d0cedc009ee5887cc68f1f3.woff
-rw-rw-r-- 1 kibana kibana     9707 Dec 27 17:03 commons.style.css
total 53848

Just stopping and starting the kibana service gave the same error.

I did;

• rm -rf /usr/share/kibana/optimize/*
• and then started the service
• and got the message in the log "message":"Optimizing and caching bundles for ml, stateSessionStorageRedirect, status_page, timelion, graph, monitoring, login, logout, dashboardViewer, apm and kibana. This may take a few minutes"}
• and it started successfully.

[LeeDr]

I just hit this again on 6.2.0-SNAPSHOT rpm package on CentOS. In this case I didn't rm the optimize directory, I just started the service again and it worked.

[LeeDr]

Here's the ls -laR output of the /usr/share/kibana/optimize/bundles directory on CentOS in a case where I didn't have the permission error (6.2.0-SNAPSHOT again);

centOSbundles.txt

[LeeDr]

I did also hit the same problem on ubuntu with the .deb package on 6.2.0-SNAPSHOT;

{"type":"error","@timestamp":"2018-01-05T21:45:08Z","tags":["fatal"],"pid":3579,"level":"fatal","error":{"message":"EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/ml.entry.js'","n ame":"Error","stack":"Error: EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/ml.entry.js'\n at Error (native)","code":"EACCES"},"message":"EACCES: permission denied, open '/usr/ share/kibana/optimize/bundles/ml.entry.js'"}

[LeeDr]

I'm going to mark this as a blocker for 6.2. It seems like something changed from 6.1.x to make this worse. Even if we document to install x-pack as the kibana user I think we need to change how that kibana user is created in 6.2 otherwise it doesn't work (the user is created with --no-login or something like that).

[tylersmalley]

The snapshots were last created on January 3rd. It appears this is resolved by https://github.com/elastic/kibana/pull/15910 for 6.2. Per the documentation, we should be using sudo -u kibana when installing Linux packages to avoid permission issues. We might need to link to this from the x-pack documentation to make this more known.

Here are current builds:

https://download.elastic.co/kibana/staging/6.2.0-SNAPSHOT-9578180/kibana/kibana-6.2.0-SNAPSHOT-amd64.deb https://download.elastic.co/kibana/staging/6.2.0-SNAPSHOT-9578180/kibana/kibana-6.2.0-SNAPSHOT-darwin-x86_64.tar.gz https://download.elastic.co/kibana/staging/6.2.0-SNAPSHOT-9578180/kibana/kibana-6.2.0-SNAPSHOT-linux-x86_64.tar.gz https://download.elastic.co/kibana/staging/6.2.0-SNAPSHOT-9578180/kibana/kibana-6.2.0-SNAPSHOT-windows-x86_64.zip https://download.elastic.co/kibana/staging/6.2.0-SNAPSHOT-9578180/kibana/kibana-6.2.0-SNAPSHOT-x86_64.rpm

[sophiec20]

@jimgoodwin FF tomorrow. Is this a blocker for 6.2.2?

[tylersmalley]

@LeeDr, would you mind giving this another look. The issue you brought up when adding the blocker should be resolved from the merge of https://github.com/elastic/kibana/pull/15910

[LeeDr]

I think there's two issues;

1. In #15910 @spalger fixed the sorting issue so we didn't hit an Optimize when we shouldn't. I do believe this is fixed (at least on 6.2 branch where I'm currently testing the latest snapshot in preparation for a 6.2.2 patch).

2. Permission denied errors when something changes that does require Optimize to run again. I'm currently installing the Kibana x-pack as the Kibana user with sudo -u kibana and I think that eliminates the permission denied errors. I thought we were going to update the x-pack kibana installation steps with that? I don't see it updated here yet; https://www.elastic.co/guide/en/kibana/6.2/installing-xpack-kb.html

[ddrake17]

+1 - just saw this after installing x-pack on 5.6.8

[zhexiao]

mark, i have the same problem on KIBANA 6.2 :+1:

[tylersmalley]

@ddrake17 @zhexiao did you install any plugins as the kibana user?

[LeeDr]

@zhexiao was your problem specifically on 6.2.0? Or later? There was a change in 6.2.2 (latest release is 6.2.3 so that also has the fix).

Also, please describe which package you're using (.deb, .rpm, .tar.gz) and if you installed x-pack, and exactly how you installed x-pack (including what user).

If you untar the tar.gz package of Kibana as a non-root user. And also install x-pack as a non-root user I think there's no problem. Please correct me if I'm wrong.

EVERYONE

If you install the .deb or .rpm package you have to do that as root. But you should install x-pack as the kibana user (sudo -u kibana) to avoid the permission problems.

[LeeDr]

I'm going to close this since I believe following the lastest docs for installing x-pack resolves this issue on current releases.

[aguyinmontreal]

kibana x-pack uninstall (version 6.2.4) is still leaving leftovers in the optimize/bundles directory. @jbudz 's solution is still a necessary step before a kibana x-pack reinstall

mv optimize/bundles optimize/old_bundles sudo -u kibana bin/kibana-plugin install x-pack

[LeeDr]

@aguyinmontreal from 6.3+ you can't install or uninstall x-pack. It's built-in to the default distribution.

There's a separate OSS distribution without x-pack, but since we're no longer packing x-pack as a plugin, you can't install it there either.

[aguyinmontreal]

@LeeDr

So everyone gets the trial license installed by default now? How is it possible to install X-pack with the "basic" license instead of the "trial" license?

Before we were able to set xpack.license.self_generated.type: basic  in elasticsearch.yml. But now, there's no elasticsearch.yml since X-pack is installed at the same time as Elasticsearch.

[epixa]

@aguyinmontreal The default distribution now ships with a basic license out of the box for everyone. Folks can choose to opt-in to a trial via the Kibana license management UI.