search

elastic / elasticsearch-ruby

Ruby integrations for Elasticsearch
https://www.elastic.co/guide/en/elasticsearch/client/ruby-api/current/index.html
Apache License 2.0
1.97k stars 597 forks source link

Authorization Service not Available #294

Closed AaronRustad closed 8 years ago

AaronRustad commented 8 years ago

We're using AWS Elasticsearch service and we're currently seeing a number of the following errors:

Elasticsearch::Transport::Transport::Errors::ServiceUnavailable: [503] {"Message":"Call to Authorization engine failed"}

The number of requests to ES is usually around 3000/day, and we've been seeing a couple every one or two days (today we saw 10). We issuing these requests from a Rails application and the following code is how we initialize the client.

require 'elasticsearch'
require 'elasticsearch/model'
require 'faraday_middleware/aws_signers_v4'

def create_client(hosts)
  config = {
               hosts: hosts,
              logger: ActiveSupport::Logger.new("#{Rails.root}/log/elasticsearch.log"),
    retry_on_failure: true,
   reload_on_failure: true,
     sniffer_timeout: 3,
     request_timeout: 5
  }

  case Rails.env
  when 'development', 'staging'
    config.merge!(trace: true) unless 'rake' == File.basename($0)
  end

  Elasticsearch::Client.new(config) do |f|
    if Rails.env.production?
      # We should be able to remove this once we move into the VPC
      f.request(:aws_signers_v4, credentials: Aws::Credentials.new(AppConfig.aws_access_key_id, AppConfig.aws_secret_access_key), service_name: 'es', region: 'us-east-1')
    end
  end
end

es_hosts         = Array(ENV.fetch('ELASTICSEARCH_URL', AppConfig.es_hosts))
es_context_hosts = Array(ENV.fetch('ELASTICSEARCH_CONTEXT_URL', AppConfig.es_context_hosts))

Elasticsearch::Model.client = create_client(es_hosts)
Elasticsearch::Persistence.client = create_client(es_context_hosts)

Of particular interest is the fact that we use two Elasticsearch clients (two AWS ES clusters). I'm not sure if that could be contributing to the issue. If the signing of requests tied to the service endpoint perhaps, perhaps two clients will result in the occasional poorly worded error?

Stack trace follows:

/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/base.rb:146 in __raise_transport_error
/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/base.rb:256 in perform_request
/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/http/faraday.rb:20 in perform_request
/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/client.rb:125 in perform_request
/gems/elasticsearch-api-1.0.15/lib/elasticsearch/api/actions/search.rb:167 in search
/gems/elasticsearch-model-0.1.8/lib/elasticsearch/model/searching.rb:51 in execute!
/gems/elasticsearch-model-0.1.8/lib/elasticsearch/model/response.rb:31 in response
/gems/elasticsearch-model-0.1.8/lib/elasticsearch/model/response/records.rb:34 in ids
/gems/elasticsearch-model-0.1.8/lib/elasticsearch/model/adapters/active_record.rb:16 in records
/gems/elasticsearch-model-0.1.8/lib/elasticsearch/model/response/records.rb:58 in method_missing
/data/f/srv/client_portal/releases/20160323195123/app/controllers/api/v1/market_contents_controller.rb:11 in block in search
/data/f/srv/client_portal/releases/20160323195123/app/controllers/api/v1/market_contents_controller.rb:30 in process_criteria
/data/f/srv/client_portal/releases/20160323195123/app/controllers/api/v1/market_contents_controller.rb:9 in search
/gems/actionpack-4.2.6/lib/action_controller/metal/implicit_render.rb:4 in send_action
/gems/actionpack-4.2.6/lib/abstract_controller/base.rb:198 in process_action
/gems/actionpack-4.2.6/lib/action_controller/metal/rendering.rb:10 in process_action
/gems/actionpack-4.2.6/lib/abstract_controller/callbacks.rb:20 in block in process_action
/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:117 in call
/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:555 in block (2 levels) in compile
/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:505 in call
/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:92 in __run_callbacks__
/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:778 in _run_process_action_callbacks
/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:81 in run_callbacks
/gems/actionpack-4.2.6/lib/abstract_controller/callbacks.rb:19 in process_action
/gems/actionpack-4.2.6/lib/action_controller/metal/rescue.rb:29 in process_action
/gems/actionpack-4.2.6/lib/action_controller/metal/instrumentation.rb:32 in block in process_action
/gems/activesupport-4.2.6/lib/active_support/notifications.rb:164 in block in instrument
/gems/activesupport-4.2.6/lib/active_support/notifications/instrumenter.rb:20 in instrument
/gems/activesupport-4.2.6/lib/active_support/notifications.rb:164 in instrument
/gems/actionpack-4.2.6/lib/action_controller/metal/instrumentation.rb:30 in process_action
/gems/actionpack-4.2.6/lib/action_controller/metal/params_wrapper.rb:250 in process_action
/gems/activerecord-4.2.6/lib/active_record/railties/controller_runtime.rb:18 in process_action
/gems/actionpack-4.2.6/lib/abstract_controller/base.rb:137 in process
/gems/actionview-4.2.6/lib/action_view/rendering.rb:30 in process
/gems/actionpack-4.2.6/lib/action_controller/metal.rb:196 in dispatch
/gems/actionpack-4.2.6/lib/action_controller/metal/rack_delegation.rb:13 in dispatch
/gems/actionpack-4.2.6/lib/action_controller/metal.rb:237 in block in action
/gems/actionpack-4.2.6/lib/action_dispatch/routing/route_set.rb:74 in dispatch
/gems/actionpack-4.2.6/lib/action_dispatch/routing/route_set.rb:43 in serve
/gems/actionpack-4.2.6/lib/action_dispatch/routing/mapper.rb:49 in serve
/gems/actionpack-4.2.6/lib/action_dispatch/journey/router.rb:43 in block in serve
/gems/actionpack-4.2.6/lib/action_dispatch/journey/router.rb:30 in each
/gems/actionpack-4.2.6/lib/action_dispatch/journey/router.rb:30 in serve
/gems/actionpack-4.2.6/lib/action_dispatch/routing/route_set.rb:817 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/rack/agent_hooks.rb:30 in traced_call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/rack-cache-1.6.1/lib/rack/cache/context.rb:140 in forward
/gems/rack-cache-1.6.1/lib/rack/cache/context.rb:249 in fetch
/gems/rack-cache-1.6.1/lib/rack/cache/context.rb:189 in lookup
/gems/rack-cache-1.6.1/lib/rack/cache/context.rb:66 in call!
/gems/rack-cache-1.6.1/lib/rack/cache/context.rb:51 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/rack-jsonp-1.3.1/lib/rack/jsonp.rb:36 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/rack-1.6.4/lib/rack/etag.rb:24 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/rack-1.6.4/lib/rack/conditionalget.rb:25 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/rack-1.6.4/lib/rack/head.rb:13 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/actionpack-4.2.6/lib/action_dispatch/middleware/params_parser.rb:27 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
[GEM_ROOT]/bundler/gems/argus-01654672c8e1/lib/argus/event.rb:111 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/warden-1.2.6/lib/warden/manager.rb:35 in block in call
/gems/warden-1.2.6/lib/warden/manager.rb:34 in catch
/gems/warden-1.2.6/lib/warden/manager.rb:34 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/actionpack-4.2.6/lib/action_dispatch/middleware/flash.rb:260 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:225 in context
/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:220 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/actionpack-4.2.6/lib/action_dispatch/middleware/cookies.rb:560 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/activerecord-4.2.6/lib/active_record/query_cache.rb:36 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:653 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/actionpack-4.2.6/lib/action_dispatch/middleware/callbacks.rb:29 in block in call
/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:88 in __run_callbacks__
/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:778 in _run_call_callbacks
/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:81 in run_callbacks
/gems/actionpack-4.2.6/lib/action_dispatch/middleware/callbacks.rb:27 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/mnt/srv/client_portal/current/gems/apoc/lib/apoc/geo_ip/country_lookup.rb:62 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/actionpack-4.2.6/lib/action_dispatch/middleware/remote_ip.rb:78 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/airbrake-5.2.1/lib/airbrake/rack/middleware.rb:22 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/actionpack-4.2.6/lib/action_dispatch/middleware/debug_exceptions.rb:17 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/actionpack-4.2.6/lib/action_dispatch/middleware/show_exceptions.rb:30 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/railties-4.2.6/lib/rails/rack/logger.rb:38 in call_app
/gems/railties-4.2.6/lib/rails/rack/logger.rb:22 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/request_store-1.3.0/lib/request_store/middleware.rb:9 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/actionpack-4.2.6/lib/action_dispatch/middleware/request_id.rb:21 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/rack-1.6.4/lib/rack/methodoverride.rb:22 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/rack-1.6.4/lib/rack/runtime.rb:18 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/activesupport-4.2.6/lib/active_support/cache/strategy/local_cache_middleware.rb:28 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/rack-1.6.4/lib/rack/sendfile.rb:113 in call
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/gems/railties-4.2.6/lib/rails/engine.rb:518 in call
/gems/railties-4.2.6/lib/rails/application.rb:165 in call
/gems/railties-4.2.6/lib/rails/railtie.rb:194 in public_send
/gems/railties-4.2.6/lib/rails/railtie.rb:194 in method_missing
/gems/newrelic_rpm-3.15.0.314/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96 in call
/data/f/srv/client_portal/shared/bundle/ruby/2.3.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:562 in process_client
/data/f/srv/client_portal/shared/bundle/ruby/2.3.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:658 in worker_loop
/data/f/srv/client_portal/shared/bundle/ruby/2.3.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:508 in spawn_missing_workers
/data/f/srv/client_portal/shared/bundle/ruby/2.3.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:132 in start
/gems/unicorn-5.0.1/bin/unicorn_rails:209 in <top (required)>
[GEM_ROOT]/bin/unicorn_rails:23 in load
[GEM_ROOT]/bin/unicorn_rails:23 in <main>

All ideas welcomed! Aaron.

MetricMike commented 8 years ago

I have a similar setup (much lower volume though), and running into the same problem.

Might be related to #220. I don't have a dedicated master node in AWS, so the 503s might simply be 2 nodes disagreeing. Whatever the cause is, and I suspect it's something internal to Amazon, I've added the retry_on_status: [502,503,504] to my client config and the errors have disappeared.

It's only been an hour since I added that setting, so I don't know if this is The Fix™.

AaronRustad commented 8 years ago

It doesn't pinpoint the problem, but knowing that AWS rarely explains any underlying problems, it might be a wise thing to do the same.

Thanks.

karmi commented 8 years ago

@AaronRustad, unfortunately, it doesn't ring any bells... As @MetricMike suggests, are you using connection reloading ("sniffing")? The AWS setup might trip ut the process, but I haven't investigated it in a real environment.

karmi commented 8 years ago

Hi @AaronRustad, any news? :)

AaronRustad commented 8 years ago

Hi! I did use @MetricMike's retry_on_status and we're not seeing the errors. To be fair, we might still be getting them, but now they're hidden by the retry. However, it seems to be a decent solution and I don't have any evidence that it is a multi-client issue.

karmi commented 8 years ago

I see. The errors should stil be in the logs, so if it looks like a problem, ping me again please!

gregblass commented 6 years ago

I just had this error happen on a client's production server. Any insight as to what might have caused this?

SeanRoy commented 4 years ago

I just saw this today in our production environment. US-EAST-1

picandocodigo commented 4 years ago

The Elasticsearch clients go through rigorous testing and validation against the official Elasticsearch distribution coming from Elastic. Since the AWS Elasticsearch Service has incompatible APIs with Elasticsearch itself, either by missing APIs or has incompatible format, we do not support it. Please consider using Elastic Cloud for an official and formally supported hosted Elasticsearch distribution.

gregblass commented 4 years ago

Yeah this is the conclusion I came to as well.

Not to mention the fact that amazon elasticsearch support is horrible.

There’s no reason to use it whatsoever.

On Wed, Jun 10, 2020 at 6:34 PM Fernando Briano notifications@github.com wrote:

The Elasticsearch clients go through rigorous testing and validation against the official Elasticsearch distribution coming from Elastic. Since the AWS Elasticsearch Service has incompatible APIs with Elasticsearch itself, either by missing APIs or has incompatible format, we do not support it. Please consider using Elastic Cloud https://cloud.elastic.co/ for an official and formally supported hosted Elasticsearch distribution.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/elastic/elasticsearch-ruby/issues/294#issuecomment-642302588, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACEK54ZY3B5MGOMAT223RLRWAC7BANCNFSM4B677JZQ .