search

ElasticHQ / elasticsearch-HQ

Monitoring and Management Web Application for ElasticSearch instances and clusters.
http://www.elastichq.org
Other
4.97k stars 532 forks source link

ElasticHQ fails on restart #396

Closed joshuacross closed 6 years ago

joshuacross commented 6 years ago

General information

Issue Description

When running python3 application.py with a freshly cloned repo the application starts as intended. However, when attempting to run the command again after stopping it I am greeted with the output below.

I'm unsure why PUT http://localhost:9200/.elastichq/data/hqsettings?refresh=true would be returning a 404.

Source Code / Logs


FROM cluster
2018-05-25 16:05:30,364          INFO    sqlalchemy.engine.base.Engine   base._execute_context:1154      ()
2018-05-25 16:05:30,712          INFO    sqlalchemy.engine.base.Engine   base._execute_context:1151      SELECT cluster.cluster_name AS cluster_cluster_name, cluster.cluster_ip AS cluster_cluster_ip, cluster.cluster_port AS cluster_cluster_port, cluster.cluster_scheme AS cluster_cluster_scheme, cluster.cluster_version AS cluster_cluster_version, cluster.cluster_username AS cluster_cluster_username, cluster.cluster_password AS cluster_cluster_password 
FROM cluster 
WHERE cluster.cluster_name = ?
2018-05-25 16:05:30,712          INFO    sqlalchemy.engine.base.Engine   base._execute_context:1154      ('elasticsearch',)
2018-05-25 16:05:31,052          INFO    sqlalchemy.engine.base.Engine   base._execute_context:1151      SELECT cluster.cluster_name AS cluster_cluster_name, cluster.cluster_ip AS cluster_cluster_ip, cluster.cluster_port AS cluster_cluster_port, cluster.cluster_scheme AS cluster_cluster_scheme, cluster.cluster_version AS cluster_cluster_version, cluster.cluster_username AS cluster_cluster_username, cluster.cluster_password AS cluster_cluster_password 
FROM cluster 
WHERE cluster.cluster_name = ?
2018-05-25 16:05:31,052          INFO    sqlalchemy.engine.base.Engine   base._execute_context:1154      ('elasticsearch',)
2018-05-25 16:05:31,052          INFO    elastichq       ClusterDBService.save_cluster:27        Cluster Exists: elasticsearch
2018-05-25 16:05:31,053          DEBUG   urllib3.util.retry      retry.from_int:200      Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2018-05-25 16:05:31,406          INFO    elasticsearch   base.log_request_success:83     GET http://localhost:9200/_cluster/health [status:200 request:0.354s]
2018-05-25 16:05:31,407          DEBUG   elasticsearch   base.log_request_success:85     > None
2018-05-25 16:05:31,407          DEBUG   elasticsearch   base.log_request_success:86     < {"cluster_name":"elasticsearch","status":"green","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":3,"active_primary_shards":149,"active_shards":437,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}
2018-05-25 16:05:31,408          INFO    elastichq       globals.get:118         Getting Cache Key: elastichq.service.HQService:get_settings|elasticsearch
2018-05-25 16:05:31,408          DEBUG   dogpile.lock    lock._enter:90          NeedRegenerationException
2018-05-25 16:05:31,408          DEBUG   dogpile.lock    lock._enter_create:123          no value, waiting for create lock
2018-05-25 16:05:31,409          DEBUG   dogpile.lock    lock._enter_create:127          value creation lock <dogpile.cache.region.CacheRegion._LockWrapper object at 0x112b0b5f8> acquired
2018-05-25 16:05:31,411          INFO    elastichq       globals.get:118         Getting Cache Key: elastichq.service.HQService:get_settings|elasticsearch
2018-05-25 16:05:31,412          DEBUG   dogpile.lock    lock._enter_create:144          Calling creation function
2018-05-25 16:05:31,412          DEBUG   urllib3.util.retry      retry.from_int:200      Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2018-05-25 16:05:31,529          WARNING         elasticsearch   base.log_request_fail:97        GET http://localhost:9200/.elastichq/data/hqsettings/_source [status:404 request:0.117s]
2018-05-25 16:05:31,529          DEBUG   elasticsearch   base.log_request_fail:105       > None
2018-05-25 16:05:31,530          DEBUG   elasticsearch   base.log_request_fail:110       < {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":".elastichq","index_uuid":"_na_","index":".elastichq"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":".elastichq","index_uuid":"_na_","index":".elastichq"},"status":404}
2018-05-25 16:05:31,530          DEBUG   urllib3.util.retry      retry.from_int:200      Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2018-05-25 16:05:31,644          WARNING         elasticsearch   base.log_request_fail:97        PUT http://localhost:9200/.elastichq/data/hqsettings?refresh=true [status:404 request:0.114s]
2018-05-25 16:05:31,645          DEBUG   elasticsearch   base.log_request_fail:105       > {"doc_id":"hqsettings","index_name":".elastichq","version":1,"doc_type":"data","store_metrics":true,"websocket_interval":5,"historic_poll_interval":300,"historic_days_to_store":7}
2018-05-25 16:05:31,645          DEBUG   elasticsearch   base.log_request_fail:110       < {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":".elastichq","index_uuid":"_na_","index":".elastichq"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":".elastichq","index_uuid":"_na_","index":".elastichq"},"status":404}
2018-05-25 16:05:31,645          DEBUG   dogpile.lock    lock._enter_create:150          Released creation lock
Traceback (most recent call last):
  File "/Users/josh.cross/Documents/workspace/elasticsearch-HQ/elastichq/service/HQService.py", line 53, in get_settings
    'doc_type'])
  File "/Users/josh.cross/Documents/workspace/elasticsearch-HQ/elastichq/vendor/elasticsearch/client/utils.py", line 76, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/Users/josh.cross/Documents/workspace/elasticsearch-HQ/elastichq/vendor/elasticsearch/client/__init__.py", line 449, in get_source
    doc_type, id, '_source'), params=params)
  File "/Users/josh.cross/Documents/workspace/elasticsearch-HQ/elastichq/vendor/elasticsearch/transport.py", line 314, in perform_request
    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "/Users/josh.cross/Documents/workspace/elasticsearch-HQ/elastichq/vendor/elasticsearch/connection/http_urllib3.py", line 163, in perform_request
    self._raise_error(response.status, raw_data)
  File "/Users/josh.cross/Documents/workspace/elasticsearch-HQ/elastichq/vendor/elasticsearch/connection/base.py", line 125, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elastichq.vendor.elasticsearch.exceptions.NotFoundError: TransportError(404, 'index_not_found_exception', 'no such index')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "application.py", line 16, in <module>
    application = create_app()
  File "/Users/josh.cross/Documents/workspace/elasticsearch-HQ/elastichq/__init__.py", line 39, in create_app
    init_connections()
  File "/Users/josh.cross/Documents/workspace/elasticsearch-HQ/elastichq/globals.py", line 70, in init_connections
    ClusterService().get_clusters()
  File "/Users/josh.cross/Documents/workspace/elasticsearch-HQ/elastichq/service/ClusterService.py", line 53, in get_clusters
    cluster.cluster_settings = HQService().get_settings(cluster.cluster_name)
  File "/Users/josh.cross/Library/Python/3.6/lib/python/site-packages/dogpile/cache/region.py", line 1250, in decorate
    should_cache_fn)
  File "/Users/josh.cross/Library/Python/3.6/lib/python/site-packages/dogpile/cache/region.py", line 844, in get_or_create
    async_creator) as value:
  File "/Users/josh.cross/Library/Python/3.6/lib/python/site-packages/dogpile/lock.py", line 154, in __enter__
    return self._enter()
  File "/Users/josh.cross/Library/Python/3.6/lib/python/site-packages/dogpile/lock.py", line 94, in _enter
    generated = self._enter_create(createdtime)
  File "/Users/josh.cross/Library/Python/3.6/lib/python/site-packages/dogpile/lock.py", line 145, in _enter_create
    created = self.creator()
  File "/Users/josh.cross/Library/Python/3.6/lib/python/site-packages/dogpile/cache/region.py", line 811, in gen_value
    created_value = creator()
  File "/Users/josh.cross/Library/Python/3.6/lib/python/site-packages/dogpile/cache/region.py", line 1246, in creator
    return fn(*arg, **kw)
  File "/Users/josh.cross/Documents/workspace/elasticsearch-HQ/elastichq/service/HQService.py", line 58, in get_settings
    self.save_settings(cluster_name)
  File "/Users/josh.cross/Documents/workspace/elasticsearch-HQ/elastichq/service/HQService.py", line 69, in save_settings
    body=body, refresh=True)
  File "/Users/josh.cross/Documents/workspace/elasticsearch-HQ/elastichq/vendor/elasticsearch/client/utils.py", line 76, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/Users/josh.cross/Documents/workspace/elasticsearch-HQ/elastichq/vendor/elasticsearch/client/__init__.py", line 302, in index
    _make_path(index, doc_type, id), params=params, body=body)
  File "/Users/josh.cross/Documents/workspace/elasticsearch-HQ/elastichq/vendor/elasticsearch/transport.py", line 314, in perform_request
    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "/Users/josh.cross/Documents/workspace/elasticsearch-HQ/elastichq/vendor/elasticsearch/connection/http_urllib3.py", line 163, in perform_request
    self._raise_error(response.status, raw_data)
  File "/Users/josh.cross/Documents/workspace/elasticsearch-HQ/elastichq/vendor/elasticsearch/connection/base.py", line 125, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elastichq.vendor.elasticsearch.exceptions.NotFoundError: TransportError(404, 'index_not_found_exception', 'no such index')
joshuacross commented 6 years ago

So it turns out the reason for this was that within elasticsearch.yaml action.auto_create_index was set to false and therefore was preventing ElasticHQ from creating a .elastichq index. By default it is set to true.

royrusso commented 6 years ago

Interesting. This must've changed since ES 2.x. I'll add a safe catch for this 404.