search

elasticfence / elasticsearch-http-user-auth

HTTP Basic Authentication and IP ACL plugin for Elasticsearch :key: (UNMAINTAINED)
http://qxip.net
148 stars 32 forks source link

Support of Elasticsearch 2.2 and Kibana 4.4 #2

Closed damhau closed 8 years ago

damhau commented 8 years ago

I'm trying to use the plugin with ES 2.2 and Kibana 4.4

Everything seems to work fine with the exception of Kibana 4.4, I’ve got the following error in the Kibana log when I’m login with a user that I’ve created :

respons [20:28:32.506]  GET /bundles/src/ui/public/images/kibana.svg 304 3ms - 9.0B
respons [20:28:32.508]  GET /bundles/src/ui/public/images/elk.ico 304 5ms - 9.0B
respons [20:28:32.523]  POST /elasticsearch/_mget?timeout=0&ignore_unavailable=true&preference=1455046102682 401 5ms - 9.0B
respons [20:28:32.549]  GET /bundles/src/ui/public/images/elk.ico 200 4ms - 9.0B
respons [20:28:32.549]  GET /bundles/src/ui/public/images/elk.ico 200 3ms - 9.0B
  log   [20:28:32.612] [debug][plugin] Checking Elasticsearch version
  log   [20:28:35.124] [debug][plugin] Checking Elasticsearch version
  ops   [20:28:36.826]  memory: 71.2MB uptime: 0:00:51 load: [0.01 0.02 0.05] delay: 0.389
  log   [20:28:37.647] [debug][plugin] Checking Elasticsearch version
respons [20:28:38.287]  POST /elasticsearch/_mget?timeout=0&ignore_unavailable=true&preference=1455046102682 403 4ms - 9.0B
respons [20:28:38.598]  GET /bundles/node_modules/font-awesome/fonts/fontawesome-webfont.woff2 304 2ms - 9.0B

If I login with the root user it works, see the log of a sucessfull connexion :

respons [20:28:02.520]  POST /elasticsearch/.kibana/config/4.3.1/_update 200 31ms - 9.0B
respons [20:28:03.046]  POST /elasticsearch/_mget?timeout=0&ignore_unavailable=true&preference=1455046060281 200 18ms - 9.0B
respons [20:28:03.089]  POST /elasticsearch/.kibana/index-pattern/netflow-* 200 67ms - 9.0B
  log   [20:28:04.928] [debug][plugin] Checking Elasticsearch version
respons [20:28:05.430]  POST /elasticsearch/.kibana/index-pattern/_search?fields= 200 7ms - 9.0B
respons [20:28:05.783]  POST /elasticsearch/netflow-*/_field_stats?level=indices 200 7ms - 9.0B
respons [20:28:05.870]  POST /elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1455046060281 200 3ms - 9.0B
  ops   [20:28:06.820]  memory: 74.4MB uptime: 0:00:21 load: [0.02 0.02 0.05] delay: 0.487
  log   [20:28:07.461] [debug][plugin] Checking Elasticsearch version
respons [20:28:09.077]  POST /elasticsearch/netflow-*/_field_stats?level=indices 200 5ms - 9.0B
respons [20:28:09.717]  POST /elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1455046060281 200 550ms - 9.0B
  log   [20:28:09.973] [debug][plugin] Checking Elasticsearch version
  ops   [20:28:11.821]  memory: 75.5MB uptime: 0:00:26 load: [0.02 0.02 0.05] delay: 0.464
  log   [20:28:12.487] [debug][plugin] Checking Elasticsearch version
  log   [20:28:15.011] [debug][plugin] Checking Elasticsearch version

My understanding is that Kibana is sending a POST request to /_mget?timeout=0&ignore_unavailable=true&preference=1455046102682 with the following json {"docs":[{"_index":".kibana","_type":"config","_id":"4.4.0"}]} and I’m assuming that your plugin doesn’t allow this request.

I’ve confirmed with curl that my user has access to the .kibana index :

curl http://dhr:1234@10.200.99.92:9200/.kibana

{".kibana":{"aliases":{},"mappings":{"dashboard":{"properties":{"description":{"type":"string"},"hits":{"type":"integer"},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"string"}}},"optionsJSON":{"type":"string"},"panelsJSON":{"type":"string"},"timeFrom":{"type":"string"},"timeRestore":{"type":"boolean"},"timeTo":{"type":"string"},"title":{"type":"string"},"uiStateJSON":{"type":"string"},"version":{"type":"integer"}}},"visualization":{"properties":{"description":{"type":"string"},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"string"}}},"savedSearchId":{"type":"string"},"title":{"type":"string"},"uiStateJSON":{"type":"string"},"version":{"type":"integer"},"visState":{"type":"string"}}},"index-pattern":{"properties":{"fieldFormatMap":{"type":"string"},"fields":{"type":"string"},"intervalName":{"type":"string"},"timeFieldName":{"type":"string"},"title":{"type":"string"}}},"config":{"properties":{"buildNum":{"type":"string","index":"not_analyzed"},"defaultIndex":{"type":"string"}}},"search":{"properties":{"columns":{"type":"string"},"description":{"type":"string"},"hits":{"type":"integer"},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"string"}}},"sort":{"type":"string"},"title":{"type":"string"},"version":{"type":"integer"}}}},"settings":{"index":{"creation_date":"1454537508624","number_of_shards":"1","number_of_replicas":"1","uuid":"Qkf4LIW8Q5iL84xLSqAqfg","version":{"created":"2020099"}}},"warmers":{}}}

If I try with curl here are the result :

With the root user:

curl -XPOST 'http://root:toto@10.200.99.92:9200/_mget' -d '{"docs":[{"_index":".kibana","_type":"config","_id":"4.4.0"}]}'

{"docs":[{"_index":".kibana","_type":"config","_id":"4.4.0","_version":2,"found":true,"_source":{"buildNum":9689,"defaultIndex":"netflow-*"}}]}

With a standard user:

curl -XPOST 'http://dhr:1234@10.200.99.92:9200/_mget' -d '{"docs":[{"_index":".kibana","_type":"config","_id":"4.4.0"}]}'

Forbidden path

The new version of Kibana is using /_msearch for every query so the index is not part of the URL path anymore.

I think that the way to implement the support of Kibana 4.4 is to match the content of the index field in the post request.

I've found an example here : https://templth.wordpress.com/2014/03/07/implementing-data-validation-in-elasticsearch/ but the java part is way above my head.

Do you think there is a way to implement somehting like this ?

Ps: I'm available if you need anything else to help you to solve the problem.

tommyktech commented 8 years ago

Thank you for your report!

Your understanding is correct. This plugin just checks URL paths, not request bodies right now. While I searched around the Elasticsearch repo for a lazy and efficient solution to this problem, it seems to be better to pick up index fields from POST request bodies, as you said. (In fact, Elasticsearch breaks down a _msearch request into single search requests internally, so I tried to cut into the process for checking indices, but I found it impossible finally. )

When I walked around the ES repo, I found a function that parses _msearch request. https://github.com/elastic/elasticsearch/blob/ba29818629861240a5971c31849ddf405a0a831d/core/src/main/java/org/elasticsearch/rest/action/search/RestMultiSearchAction.java#L104 Using this code, it would be easy to create a simple parser to extract indices from _msearch request. I'll try it :octocat:

In addition to _msearch, _bulk has the same API format as far as I know. Please let me know if you know other APIs that have index fields in their request bodies :octocat: :octocat:

tommyktech commented 8 years ago

I created a new branch 2.2.x. With this version, you can use kibana 4.4 with the user you created.

Preparation steps:

  1. Install this version (bin/plugin install https://raw.githubusercontent.com/elasticfence/elasticsearch-http-user-auth/2.2.x/jar/elasticfence-2.2.0-SNAPSHOT.zip)
  2. Add an index filter "/.kibana" to an user which you created on Elasticfence and you set in kibana.yml
  3. Access to your Kibana instance

Of course it's needed to add index filters you want to access on Kibana

In detail, the changes are:

  1. Added request body parsers to some APIs (_mget, _msearch, and _bulk)
  2. With "/.kibana" index filter, this plugin allows the user to access /_nodes, /_cluster/health/.kibana additionally so that Kibana works normally.

I'm looking forward to your feedback!

lmangani commented 8 years ago

+1

damhau commented 8 years ago

Wow you re fast !!! That s great !!! I ll test as soon as possible and let you know.

Thank you very much you re the best :-)

Sent from my iPhone

On 14 Feb 2016, at 17:59, Tom Search notifications@github.com wrote:

I created a new branch 2.2.x. With this version, you can use kibana 4.4 with the user you created.

Preparation steps:

  1. Install this version (bin/plugin install https://raw.githubusercontent.com/elasticfence/elasticsearch-http-user-auth/2.2.x/jar/elasticfence-2.2.0-SNAPSHOT.zip)
  2. Add an index filter "/.kibana" to an user which you created on Elasticfence and you set in kibana.yml
  3. Access to your Kibana instance

Of course it's necessary to add index filters you want to access on Kibana

In detail, the changes are:

  1. Added request body parsers to some APIs (_mget, _msearch, and _bulk)
  2. With "/.kibana" index filter, this plugin allows the user to access /_nodes, /_cluster/health/.kibana additionally so that Kibana works normally.

I'm looking forward to your feedback!

— Reply to this email directly or view it on GitHub.

damhau commented 8 years ago

I’m testing the new version and I’ve a small problem, it seems that the config of the root user password is not working anymore.

Here is what I got :

[root@cust01-es01 ~]# /usr/share/elasticsearch/bin/plugin install https://raw.githubusercontent.com/elasticfence/elasticsearch-http-user-auth/2.2.x/jar/elasticfence-2.2.0-SNAPSHOT.zip -> Installing from https://raw.githubusercontent.com/elasticfence/elasticsearch-http-user-auth/2.2.x/jar/elasticfence-2.2.0-SNAPSHOT.zip... Trying https://raw.githubusercontent.com/elasticfence/elasticsearch-http-user-auth/2.2.x/jar/elasticfence-2.2.0-SNAPSHOT.zip ... Downloading .................................DONE Verifying https://raw.githubusercontent.com/elasticfence/elasticsearch-http-user-auth/2.2.x/jar/elasticfence-2.2.0-SNAPSHOT.zip checksums if available ... NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify) Installed elasticfence into /usr/share/elasticsearch/plugins/elasticfence [root@cust01-es01 ~]#

cat /etc/elasticsearch/elasticsearch.yml

http.jsonp.enable: true

http.user.auth.disabled: false http.user.auth.root.password: ***** http.user.auth.whitelist: ["127.0.0.1", "10.200.101.41", "10.200.99.95", "10.200.254.2"] [root@cust01-es01 ~]#

[root@cust01-es01 ~]# service elasticsearch restart Restarting elasticsearch (via systemctl): [ OK ]

tail -100 /var/log/elasticsearch/es-cluster.log

[2016-02-17 11:23:10,321][INFO ][node ] [cust01-es01] version[2.2.0], pid[23998], build[8ff36d1/2016-01-27T13:32:39Z] [2016-02-17 11:23:10,321][INFO ][node ] [cust01-es01] initializing ... [2016-02-17 11:23:10,876][INFO ][plugin.elasticfence ] loading elasticfence plugin... [2016-02-17 11:23:10,876][INFO ][plugins ] [cust01-es01] modules [lang-expression, lang-groovy], plugins [head, elasticfence], sites [head, elasticfence] [2016-02-17 11:23:10,893][INFO ][env ] [cust01-es01] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [34.4gb], net total_space [37.4gb], spins? [unknown], types [rootfs] [2016-02-17 11:23:10,893][INFO ][env ] [cust01-es01] heap size [1015.6mb], compressed ordinary object pointers [true] [2016-02-17 11:23:11,040][INFO ][plugin.elasticfence ] elasticfence.disabled: null [2016-02-17 11:23:11,040][WARN ][plugin.elasticfence ] elasticfence plugin IP whitelist enabled [127.0.0.1] [2016-02-17 11:23:11,040][WARN ][plugin.elasticfence ] elasticfence plugin IP blacklist enabled [] [2016-02-17 11:23:11,040][INFO ][plugin.elasticfence ] elasticfence plugin is enabled [2016-02-17 11:23:12,613][INFO ][node ] [cust01-es01] initialized [2016-02-17 11:23:12,614][INFO ][node ] [cust01-es01] starting ...

[2016-02-17 11:23:12,766][INFO ][discovery ] [cust01-es01] es-cluster/Vjb60qASRiqinVHBaPqR3A [2016-02-17 11:23:15,827][INFO ][cluster.service ] [cust01-es01] new_master {cust01-es01}{Vjb60qASRiqinVHBaPqR3A}{10.200.99.92}{10.200.99.92:9300}{master=true}, added {{cust01-kib01}{csEQP2nkQVeXqW_QG-7r7A}{10.200.99.95}{10.200.99.95:9300}{data=false, master=false},}, reason: zen-disco-join(elected_as_master, [0] joins received)

[2016-02-17 11:23:15,932][INFO ][node ] [cust01-es01] started [2016-02-17 11:23:16,136][INFO ][gateway ] [cust01-es01] recovered [23] indices into cluster_state [2016-02-17 11:23:17,071][INFO ][plugin.elasticfence ] Invalid User: /_bulk [2016-02-17 11:23:19,073][INFO ][plugin.elasticfence ] Invalid User: /_bulk [2016-02-17 11:23:20,536][INFO ][cluster.routing.allocation] [cust01-es01] Cluster health status changed from [RED] to [YELLOW](reason: [shards started [[netflow-03.02.2016][3], [netflow-03.02.2016][1], [.kibana][0]] ...]). [2016-02-17 11:23:21,076][INFO ][plugin.elasticfence ] Invalid User: /_bulk [2016-02-17 11:23:23,079][INFO ][plugin.elasticfence ] Invalid User: /_bulk [2016-02-17 11:23:25,082][INFO ][plugin.elasticfence ] Invalid User: /_bulk [2016-02-17 11:23:27,085][INFO ][plugin.elasticfence ] Invalid User: /_bulk [2016-02-17 11:23:29,087][INFO ][plugin.elasticfence ] Invalid User: /_bulk [2016-02-17 11:23:31,089][INFO ][plugin.elasticfence ] Invalid User: /_bulk [2016-02-17 11:23:33,092][INFO ][plugin.elasticfence ] Invalid User: /_bulk [2016-02-17 11:23:35,094][INFO ][plugin.elasticfence ] Invalid User: /_bulk [2016-02-17 11:23:37,096][INFO ][plugin.elasticfence ] Invalid User: /_bulk [2016-02-17 11:23:39,098][INFO ][plugin.elasticfence ] Invalid User: /_bulk [2016-02-17 11:23:41,100][INFO ][plugin.elasticfence ] Invalid User: /_bulk [2016-02-17 11:23:43,103][INFO ][plugin.elasticfence ] Invalid User: /_bulk [2016-02-17 11:23:45,105][INFO ][plugin.elasticfence ] Invalid User: /_bulk [2016-02-17 11:23:47,108][INFO ][plugin.elasticfence ] Invalid User: /_bulk [2016-02-17 11:23:49,110][INFO ][plugin.elasticfence ] Invalid User: /_bulk

[root@cust01-es01 ~]# curl http://root:garfield@localhost:9200 Needs Basic Auth [root@cust01-es01 ~]#

[root@cust01-es01 ~]# curl localhost:9200 Needs Basic Auth [root@cust01-es01 ~]#

Maybe I’m not completely awake and I missed somehting obvious, it seems that neither the http.user.auth.whitelist settings neither the http.user.auth.root.password: settings are parsed correctly.

I don’t have the time right now to add some debug in the java code to understand what’s going on, I’ll try later.

In the meantime if you have any idea ?

lmangani commented 8 years ago

Investigating - mind localhost != 127.0.0.1 necessarily and that is what you whitelisted. Could you try with curl http://root:*****@127.0.0.1:9200 to confirm the second issue?

Edit: the output might be misleading but actually indicates the path not being allowed for the specific user. I would suggest trying both 127.0.0.1 and create a user with explicit access to all paths (or clarify if this is already the case) - Thanks!

tommyktech commented 8 years ago

Please replace the config prefix string "http.user.auth." for "elasticfence.". That is to say, for instance, the configs were : 

http.user.auth.disabled: false
http.user.auth.root.password: ~~~
http.user.auth.whitelist: [~~~]

but now, we should write as below: 

elasticfence.disabled: false
elasticfence.root.password: ~~~
elasticfence.whitelist: [~~~]
elasticfence.blacklist: [~~~]

(Please see also the 2.2.x's README => https://github.com/elasticfence/elasticsearch-http-user-auth/tree/2.2.x#configuration)

This is because we are now changing plugin's name from http-user-auth to Elasticfence. I think I had better change the Java code to accept both prefixes and output some warning (deprecated) log for the present.

damhau commented 8 years ago

Alright, I should have seen the code that parse the config file (I m a looser :-)).

Yeah I think updating the java code to accept both would be a good idea...

Sent from my iPhone

On 17 Feb 2016, at 16:18, Tom Search notifications@github.com wrote:

Please replace the config prefix string "http.user.auth." for "elasticfence.". That is to say, for instance, the configs were :

http.user.auth.disabled: false http.user.auth.root.password: ~~~ http.user.auth.whitelist: [~~~] but now, we should write as below:

elasticfence.disabled: false elasticfence.root.password: ~~~ elasticfence.whitelist: [~~~] elasticfence.blacklist: [~~~] This is because we are now changing plugin's name from http-user-auth to Elasticfence. I think I had better change the Java code to accept both prefixes, and output some warning (depricated) log for the present.

https://github.com/elasticfence/elasticsearch-http-user-auth/tree/2.2.x

— Reply to this email directly or view it on GitHub.

damhau commented 8 years ago

So I've changed the elasticsearch config and it look better but still not working 100%.

I've created a user with the following index /esxi-,/netflow-,/.kibana

I can do a curl damien:xxx@10.200.99.92:9200/.kibana and a curl damien:xxx@10.200.99.92:9200/netflow-*

But when I login to Kibana with this user i get the following error:

Discover: Authorization Exception: Error: Authorization Exception ErrorAbstract@http://10.200.99.95:5601/bundles/kibana.bundle.js?v=9689:74976:20 StatusCodeError@http://10.200.99.95:5601/bundles/kibana.bundle.js?v=9689:75116:8 respond@http://10.200.99.95:5601/bundles/kibana.bundle.js?v=9689:76343:16 checkRespForFailure@http://10.200.99.95:5601/bundles/kibana.bundle.js?v=9689:76306:8 [24]</AngularConnector.prototype.request/<@http://10.200.99.95:5601/bundles/kibana.bundle.js?v=9689:74924:8 processQueue@http://10.200.99.95:5601/bundles/commons.bundle.js?v=9689:42357:29 scheduleProcessQueue/<@http://10.200.99.95:5601/bundles/commons.bundle.js?v=9689:42373:28 $RootScopeProvider/this.$get</Scope.prototype.$eval@http://10.200.99.95:5601/bundles/commons.bundle.js?v=9689:43601:17 $RootScopeProvider/this.$get</Scope.prototype.$digest@http://10.200.99.95:5601/bundles/commons.bundle.js?v=9689:43412:16 $RootScopeProvider/this.$get</Scope.prototype.$apply@http://10.200.99.95:5601/bundles/commons.bundle.js?v=9689:43709:14 done@http://10.200.99.95:5601/bundles/commons.bundle.js?v=9689:38158:37 completeRequest@http://10.200.99.95:5601/bundles/commons.bundle.js?v=9689:38356:8 requestLoaded@http://10.200.99.95:5601/bundles/commons.bundle.js?v=9689:38297:1

If I login with the root account to Kibana it works.

damhau commented 8 years ago

There was a copy paste error in my last message, the user is configure with the index /esxi-*,/netflow-*,/.kibana

tommyktech commented 8 years ago

The error message can come up on top of the Kibana UI when you choose another index which the Kibana user doesn't have permission. If you add your indices on "Indices" page (from "Settings" tab), the message may disappear. Please try it.

kibana

kibana2

damhau commented 8 years ago

I’ve alrdeay tried to add all the index I’m aware of… I’ll continue to test.

From: Tom Search [mailto:notifications@github.com] Sent: dimanche 21 février 2016 18:11 To: elasticfence/elasticsearch-http-user-auth elasticsearch-http-user-auth@noreply.github.com Cc: damhau damien.hauser@gmail.com Subject: Re: [elasticsearch-http-user-auth] Support of Elasticsearch 2.2 and Kibana 4.4 (#2)

The error message can come up on top of the Kibana UI when you choose another index which the Kibana user doesn't have permission. If you add your indices on "Indices" page (from "Settings" tab), the message may disappear. Please try it.

https://cloud.githubusercontent.com/assets/13780300/13204055/0b5ad5e6-d909-11e5-8724-3425626eeb19.png

https://cloud.githubusercontent.com/assets/13780300/13204046/d1657aa8-d908-11e5-99ea-1aebe3977f9d.png

— Reply to this email directly or view it on GitHub https://github.com/elasticfence/elasticsearch-http-user-auth/issues/2#issuecomment-186861123 . https://github.com/notifications/beacon/ANfjDDNQFICQ9fNxmJHW2cCTXpv45jLSks5pmecPgaJpZM4HXKx5.gif

tommyktech commented 8 years ago

Please try my new push on branch 2.2.x. I fixed the index filtering process.

damhau commented 8 years ago

I’m testing it right now…

From: Tom Search [mailto:notifications@github.com] Sent: dimanche 21 février 2016 19:10 To: elasticfence/elasticsearch-http-user-auth elasticsearch-http-user-auth@noreply.github.com Cc: damhau damien.hauser@gmail.com Subject: Re: [elasticsearch-http-user-auth] Support of Elasticsearch 2.2 and Kibana 4.4 (#2)

Please try my new push on branch 2.2.x. I fixed the index filtering process.

— Reply to this email directly or view it on GitHub https://github.com/elasticfence/elasticsearch-http-user-auth/issues/2#issuecomment-186875658 . https://github.com/notifications/beacon/ANfjDDJT6F9XM26olkGyC86On04wQ_oaks5pmfT5gaJpZM4HXKx5.gif

damhau commented 8 years ago

Works perfectly now !!! Thank you very much !!!

I’ll let you know if I find any bug during my tests.

From: Tom Search [mailto:notifications@github.com] Sent: dimanche 21 février 2016 19:10 To: elasticfence/elasticsearch-http-user-auth elasticsearch-http-user-auth@noreply.github.com Cc: damhau damien.hauser@gmail.com Subject: Re: [elasticsearch-http-user-auth] Support of Elasticsearch 2.2 and Kibana 4.4 (#2)

Please try my new push on branch 2.2.x. I fixed the index filtering process.

— Reply to this email directly or view it on GitHub https://github.com/elasticfence/elasticsearch-http-user-auth/issues/2#issuecomment-186875658 . https://github.com/notifications/beacon/ANfjDDJT6F9XM26olkGyC86On04wQ_oaks5pmfT5gaJpZM4HXKx5.gif

lmangani commented 8 years ago

Resolving