search

elastic / elasticsearch-ruby

Ruby integrations for Elasticsearch
https://www.elastic.co/guide/en/elasticsearch/client/ruby-api/current/index.html
Apache License 2.0
6 stars 601 forks source link

Missing authentication token when starting up Elasticsearch::Extensions::Test::Cluster #513

Closed matthinea closed 5 years ago

matthinea commented 6 years ago

In a rails_helper file I'm using to configure RSpec for my test suite, I'm running:

Elasticsearch::Extensions::Test::Cluster.start(
  timeout: 120, command: PATH_TO_ELASTIC_CMD, port: 925
)

However, the test cluster consistently fails to start, either printing the error raised by the __get_cluster_health method in cluster.rb or raising an ElasticsearchSecurityException.

Output with ENV['DEBUG'] = 'true' looks respectively like one of these:

[!] Cannot find Elasticsearch .jar from path to command [~/work/tools/elasticsearch-6.2.2/bin/elasticsearch], using `~/work/tools/elasticsearch-6.2.2/bin/elasticsearch --version`
File [~/work/tools/elasticsearch-6.2.2/bin/elasticsearch] does not exists, checking full path by `which`: "/Users/matthew/work/tools/elasticsearch-6.2.2/bin/elasticsearch"
Running [~/work/tools/elasticsearch-6.2.2/bin/elasticsearch --version] to determine version
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
> Version: 6.2.2, Build: 10b1edd/2018-02-16T19:01:30.685723Z, JVM: 9.0.4
Starting 2 Elasticsearch nodes..
Using Elasticsearch version [6.0]
~/work/tools/elasticsearch-6.2.2/bin/elasticsearch -E cluster.name=elasticsearch-test-matthews-macbook-pro.local -E node.name=node-1 -E http.port=9250 -E path.data=/tmp/elasticsearch_test -E path.logs=/tmp/log/elasticsearch -E cluster.routing.allocation.disk.threshold_enabled=false -E network.host=_local_ -E node.attr.testattr=test -E path.repo=/tmp -E repositories.url.allowed_urls=http://snapshot.test* -E discovery.zen.minimum_master_nodes=1 -E node.max_local_storage_nodes=2 -E logger.level=DEBUG 
~/work/tools/elasticsearch-6.2.2/bin/elasticsearch -E cluster.name=elasticsearch-test-matthews-macbook-pro.local -E node.name=node-2 -E http.port=9251 -E path.data=/tmp/elasticsearch_test -E path.logs=/tmp/log/elasticsearch -E cluster.routing.allocation.disk.threshold_enabled=false -E network.host=_local_ -E node.attr.testattr=test -E path.repo=/tmp -E repositories.url.allowed_urls=http://snapshot.test* -E discovery.zen.minimum_master_nodes=1 -E node.max_local_storage_nodes=2 -E logger.level=DEBUG 
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"error"=>{"root_cause"=>[{"type"=>"security_exception", "reason"=>"missing authentication token for REST request [/_cluster/health?wait_for_status=green]", "header"=>{"WWW-Authenticate"=>"Basic realm=\"security\" charset=\"UTF-8\""}}], "type"=>"security_exception", "reason"=>"missing authentication token for REST request [/_cluster/health?wait_for_status=green]", "header"=>{"WWW-Authenticate"=>"Basic realm=\"security\" charset=\"UTF-8\""}}, "status"=>401}
.Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"error"=>{"root_cause"=>[{"type"=>"security_exception", "reason"=>"missing authentication token for REST request [/_cluster/health?wait_for_status=green]", "header"=>{"WWW-Authenticate"=>"Basic realm=\"security\" charset=\"UTF-8\""}}], "type"=>"security_exception", "reason"=>"missing authentication token for REST request [/_cluster/health?wait_for_status=green]", "header"=>{"WWW-Authenticate"=>"Basic realm=\"security\" charset=\"UTF-8\""}}, "status"=>401}
.{"error"=>{"root_cause"=>[{"type"=>"security_exception", "reason"=>"missing authentication token for REST request [/_cluster/health?wait_for_status=green]", "header"=>{"WWW-Authenticate"=>"Basic realm=\"security\" charset=\"UTF-8\""}}], "type"=>"security_exception", "reason"=>"missing authentication token for REST request [/_cluster/health?wait_for_status=green]", "header"=>{"WWW-Authenticate"=>"Basic realm=\"security\" charset=\"UTF-8\""}}, "status"=>401}
.{"error"=>{"root_cause"=>[{"type"=>"security_exception", "reason"=>"missing authentication token for REST request [/_cluster/health?wait_for_status=green]", "header"=>{"WWW-Authenticate"=>"Basic realm=\"security\" charset=\"UTF-8\""}}], "type"=>"security_exception", "reason"=>"missing authentication token for REST request [/_cluster/health?wait_for_status=green]", "header"=>{"WWW-Authenticate"=>"Basic realm=\"security\" charset=\"UTF-8\""}}, "status"=>401}:

or

[!] Cannot find Elasticsearch .jar from path to command [~/work/tools/elasticsearch-6.2.2/bin/elasticsearch], using `~/work/tools/elasticsearch-6.2.2/bin/elasticsearch --version`
File [~/work/tools/elasticsearch-6.2.2/bin/elasticsearch] does not exists, checking full path by `which`: "/Users/matthew/work/tools/elasticsearch-6.2.2/bin/elasticsearch"
Running [~/work/tools/elasticsearch-6.2.2/bin/elasticsearch --version] to determine version
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
> Version: 6.2.2, Build: 10b1edd/2018-02-16T19:01:30.685723Z, JVM: 9.0.4
[2018-03-19T17:14:46,478][DEBUG][r.suppressed             ] path: /_cluster/health, params: {}
org.elasticsearch.ElasticsearchSecurityException: missing authentication token for REST request [/_cluster/health]
    at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationError(Exceptions.java:36) ~[x-pack-core-6.2.2.jar:6.2.2]
    at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.missingToken(DefaultAuthenticationFailureHandler.java:69) ~[x-pack-core-6.2.2.jar:6.2.2]
    at org.elasticsearch.xpack.security.authc.AuthenticationService$AuditableRestRequest.anonymousAccessDenied(AuthenticationService.java:597) ~[x-pack-security-6.2.2.jar:6.2.2]
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$17(AuthenticationService.java:347) [x-pack-security-6.2.2.jar:6.2.2]
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:352) [x-pack-security-6.2.2.jar:6.2.2]
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:267) [x-pack-security-6.2.2.jar:6.2.2]
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$7(AuthenticationService.java:238) [x-pack-security-6.2.2.jar:6.2.2]
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:256) [x-pack-security-6.2.2.jar:6.2.2]
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:190) [x-pack-security-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) [elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:313) [x-pack-security-6.2.2.jar:6.2.2]
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:186) [x-pack-security-6.2.2.jar:6.2.2]
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:217) [x-pack-security-6.2.2.jar:6.2.2]
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:228) [x-pack-security-6.2.2.jar:6.2.2]
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:182) [x-pack-security-6.2.2.jar:6.2.2]
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:143) [x-pack-security-6.2.2.jar:6.2.2]
    at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:96) [x-pack-security-6.2.2.jar:6.2.2]
    at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:70) [x-pack-security-6.2.2.jar:6.2.2]
    at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336) [elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-6.2.2.jar:6.2.2]

Elasticsearch is working perfectly fine in a Rails console and I can query it from the test suite as long as I make sure the client instance is passed a user and a password option matching the password I generated when I set the Elasticsearch password using x-pack, i.e:

Elasticsearch::Model.client = Elasticsearch::Client.new(
    user: ENV['DEV_ELASTICSEARCH_USER'],
    password: ENV['DEV_ELASTICSEARCH_PASSWORD']
)

I'm wondering if my inability to get a test cluster running has anything to do with the Net::HTTP GET request on line 631 of cluster.rb that does not seem to take into account authentication. The Elasticsearch docs here say:

X-Pack security works with standard HTTP basic authentication headers to authenticate users. Since Elasticsearch is stateless, this header must be sent with every request:

Authorization: Basic <TOKEN>

The <TOKEN> is computed as base64(USERNAME:PASSWORD)

Thanks!

karmi commented 6 years ago

Hi, first, I don't think I've ever used the extension with a password-protected cluster, so there might be something which needs to be changed in the code.

For a client, encoding the username and password in the URL is enough, but I'm fairly sure the extension won't take it into account — I think the extension needs additional parameters to work in this scenario.

As a note, I'm still a fan of the extension, but I think for newer versions of Elasticsearch, we should migrate into a Docker-based setup, which isolates the installation much better.

For instance, this is a single command I use to test https://github.com/elastic/elasticsearch-x-pack-ruby against an Elasticsearch cluster with the platinum X-Pack license:

docker run \
  --name elasticsearch-xpack \
  --env ELASTIC_PASSWORD=changeme \
  --publish 9260:9200 \
  --volume $(pwd)/tmp/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
  --memory 4g \
  --rm \
  docker.elastic.co/elasticsearch/elasticsearch-platinum:6.1.1
karmi commented 6 years ago

Hi @mnd-dsgn, I'm afraid I can't offer more assistance / advice at this point, is there anything more I can do?

matthinea commented 6 years ago

Sorry, I haven't had time to revisit this, we've since stubbed our own way of testing Elasticsearch functionality that just uses test_-prefixed index names and cleans up after each test. But thanks @karmi, I'll be sure to investigate Docker when we return to developing Elasticsearch features

karmi commented 6 years ago

Thanks for the note, @mnd-dsgn !

estolfo commented 5 years ago

I'm going to close this as the recommendation to use Docker for starting a cluster with platinum features is best.