Closed rgaduput closed 5 months ago
Can you provide debug output of this, please?
@untergeek Hi, I have tried below config to produce the DEBUG ouput, but the ouput did not changed. May be i have to try something else ?
elasticsearch:
client:
hosts: https://infra-elasticsearch.infra:9200
verify_certs: false
other_settings:
username: ${ES_USER}
password: ${ES_PASSWD}
logging:
loglevel: DEBUG
@untergeek had to retry with --loglevel
instead of config file, and here is the debug
For some reason Elastic search config ignored the verify_certs: false
and log contains it as true
"Elasticsearch Configuration" config: {'client': {'hosts': 'https://infra-elasticsearch.infra:9200', 'http_compress': False, 'verify_certs': True}, 'other_settings': {'master_only': False, 'skip_version_test': False, 'username': 'elastic', 'password': 'REDACTED', 'api_key': {}}}
2024-03-26 16:01:59,517 DEBUG es_client.helpers.schemacheck __init__:54 Schema: <function validfilters.<locals>.func at 0x7f79e4121120>
2024-03-26 16:01:59,517 DEBUG es_client.helpers.schemacheck __init__:58 "filters" config: [{'filtertype': 'kibana', 'exclude': True}, {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7}]
2024-03-26 16:01:59,517 DEBUG es_client.helpers.schemacheck __init__:54 Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']"), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e41204a0>, msg=None), msg=None)}
2024-03-26 16:01:59,517 DEBUG es_client.helpers.schemacheck __init__:56 "filter" config: {'filtertype': 'kibana', 'exclude': True}
2024-03-26 16:01:59,517 DEBUG curator.validators.filter_functions func:78 Filter #0: {'filtertype': 'kibana', 'exclude': True}
2024-03-26 16:01:59,517 DEBUG curator.defaults.filtertypes.age age:81 AGE FILTER = [{'direction': Any('older', 'younger', msg=None)}, {'unit': Any('seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years', msg=None)}, {'unit_count': Coerce(int, msg=None)}, {'unit_count_pattern': Any(<class 'str'>, msg=None)}, {'epoch': Any(Coerce(int, msg=None), None, msg=None)}, {'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e4123420>, msg=None), msg=None)}, {'source': Any('name', 'creation_date', 'field_stats', msg=None)}, {'stats_result': Any('min_value', 'max_value', msg=None)}, {'field': Any(<class 'str'>, msg=None)}, {'timestring': Any(None, <class 'str'>, msg=None)}]
2024-03-26 16:01:59,518 DEBUG es_client.helpers.schemacheck __init__:54 Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']"), 'direction': Any('older', 'younger', msg=None), 'unit': Any('seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years', msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, msg=None), 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e4123420>, msg=None), msg=None), 'source': Any('name', 'creation_date', 'field_stats', msg=None), 'stats_result': Any('min_value', 'max_value', msg=None), 'field': Any(<class 'str'>, msg=None), 'timestring': Any(None, <class 'str'>, msg=None)}
2024-03-26 16:01:59,518 DEBUG es_client.helpers.schemacheck __init__:56 "filter" config: {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7}
2024-03-26 16:01:59,518 DEBUG curator.validators.filter_functions func:78 Filter #1: {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7, 'stats_result': 'min_value', 'timestring': None, 'exclude': False, 'epoch': None}
2024-03-26 16:01:59,518 DEBUG es_client.helpers.schemacheck __init__:54 Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'cold2frozen', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'cold2frozen', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']")}
2024-03-26 16:01:59,518 DEBUG es_client.helpers.schemacheck __init__:56 "action type" config: {'action': 'delete_indices', 'description': 'Clean up ES by deleting indices using excess disk space', 'options': {'continue_if_exception': False, 'disable_action': False, 'ignore_empty_list': True}, 'filters': [{'filtertype': 'kibana', 'exclude': True}, {'filtertype': 'space', 'disk_space': 6, 'use_age': True, 'source': 'creation_date'}]}
2024-03-26 16:01:59,518 DEBUG es_client.helpers.schemacheck __init__:54 Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'cold2frozen', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'cold2frozen', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']"), 'description': Any(<class 'str'>, <class 'str'>, msg=None), 'options': <class 'dict'>, 'filters': <class 'list'>}
2024-03-26 16:01:59,521 DEBUG es_client.helpers.schemacheck __init__:56 "structure" config: {'action': 'delete_indices', 'description': 'Clean up ES by deleting indices using excess disk space', 'options': {'continue_if_exception': False, 'disable_action': False, 'ignore_empty_list': True}, 'filters': [{'filtertype': 'kibana', 'exclude': True}, {'filtertype': 'space', 'disk_space': 6, 'use_age': True, 'source': 'creation_date'}]}
2024-03-26 16:01:59,522 DEBUG es_client.helpers.schemacheck __init__:54 Schema: {'allow_ilm_indices': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e41374c0>, msg=None), msg=None), 'continue_if_exception': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e4135800>, msg=None), msg=None), 'disable_action': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e4135620>, msg=None), msg=None), 'ignore_empty_list': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e4135b20>, msg=None), msg=None), 'timeout_override': Any(Coerce(int, msg=None), None, msg=None)}
2024-03-26 16:01:59,522 DEBUG es_client.helpers.schemacheck __init__:56 "options" config: {'continue_if_exception': False, 'disable_action': False, 'ignore_empty_list': True}
2024-03-26 16:01:59,522 DEBUG es_client.helpers.schemacheck __init__:54 Schema: <function validfilters.<locals>.func at 0x7f79e4123600>
2024-03-26 16:01:59,522 DEBUG es_client.helpers.schemacheck __init__:58 "filters" config: [{'filtertype': 'kibana', 'exclude': True}, {'filtertype': 'space', 'disk_space': 6, 'use_age': True, 'source': 'creation_date'}]
2024-03-26 16:01:59,523 DEBUG es_client.helpers.schemacheck __init__:54 Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']"), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e41204a0>, msg=None), msg=None)}
2024-03-26 16:01:59,523 DEBUG es_client.helpers.schemacheck __init__:56 "filter" config: {'filtertype': 'kibana', 'exclude': True}
2024-03-26 16:01:59,523 DEBUG curator.validators.filter_functions func:78 Filter #0: {'filtertype': 'kibana', 'exclude': True}
2024-03-26 16:01:59,523 DEBUG es_client.helpers.schemacheck __init__:54 Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']"), 'disk_space': Any(Coerce(float, msg=None), msg=None), 'reverse': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e4122b60>, msg=None), msg=None), 'use_age': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e4122840>, msg=None), msg=None), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e4122ac0>, msg=None), msg=None), 'threshold_behavior': Any('greater_than', 'less_than', msg=None), 'source': Any('name', 'creation_date', 'field_stats', msg=None), 'stats_result': Any('min_value', 'max_value', msg=None), 'field': Any(<class 'str'>, msg=None), 'timestring': Any(None, <class 'str'>, msg=None)}
2024-03-26 16:01:59,523 DEBUG es_client.helpers.schemacheck __init__:56 "filter" config: {'filtertype': 'space', 'disk_space': 6, 'use_age': True, 'source': 'creation_date'}
2024-03-26 16:01:59,523 DEBUG curator.validators.filter_functions func:78 Filter #1: {'filtertype': 'space', 'disk_space': 6.0, 'use_age': True, 'source': 'creation_date', 'reverse': True, 'threshold_behavior': 'greater_than', 'stats_result': 'min_value', 'timestring': None, 'exclude': False}
2024-03-26 16:01:59,524 DEBUG curator.classdef __init__:49 Action Configuration: {'actions': {1: {'action': 'delete_indices', 'description': 'Clean up ES by deleting old indices', 'options': {'continue_if_exception': False, 'disable_action': False, 'ignore_empty_list': True, 'allow_ilm_indices': False, 'timeout_override': None}, 'filters': [{'filtertype': 'kibana', 'exclude': True}, {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7, 'stats_result': 'min_value', 'timestring': None, 'exclude': False, 'epoch': None}]}, 2: {'action': 'delete_indices', 'description': 'Clean up ES by deleting indices using excess disk space', 'options': {'continue_if_exception': False, 'disable_action': False, 'ignore_empty_list': True, 'allow_ilm_indices': False, 'timeout_override': None}, 'filters': [{'filtertype': 'kibana', 'exclude': True}, {'filtertype': 'space', 'disk_space': 6.0, 'use_age': True, 'source': 'creation_date', 'reverse': True, 'threshold_behavior': 'greater_than', 'stats_result': 'min_value', 'timestring': None, 'exclude': False}]}}}
2024-03-26 16:01:59,524 DEBUG curator.cli.ActionDef log_the_options:225 For action delete_indices: disable_action=Falsecontinue_if_exception=False, timeout_override=Noneignore_empty_list=True, allow_ilm_indices=False
2024-03-26 16:01:59,524 DEBUG curator.cli.ActionDef log_the_options:225 For action delete_indices: disable_action=Falsecontinue_if_exception=False, timeout_override=Noneignore_empty_list=True, allow_ilm_indices=False
2024-03-26 16:01:59,524 INFO curator.cli run:153 Preparing Action ID: 1, "delete_indices"
2024-03-26 16:01:59,524 INFO curator.cli run:160 Creating client object and testing connection
2024-03-26 16:01:59,524 DEBUG es_client.helpers.config get_client:291 Creating client object and testing connection
2024-03-26 16:01:59,525 DEBUG es_client.helpers.schemacheck __init__:54 Schema: {'other_settings': {'master_only': <function Boolean at 0x7f79e4137f60>, 'skip_version_test': <function Boolean at 0x7f79e41342c0>, 'username': Any(None, <class 'str'>, msg=None), 'password': Any(None, <class 'str'>, msg=None), 'api_key': {'id': Any(None, <class 'str'>, msg=None), 'api_key': Any(None, <class 'str'>, msg=None), 'token': Any(None, <class 'str'>, msg=None)}}, 'client': {'hosts': Any(None, <class 'list'>, <class 'str'>, msg=None), 'cloud_id': Any(None, <class 'str'>, msg=None), 'api_key': Any(None, <class 'tuple'>, msg=None), 'basic_auth': Any(None, <class 'tuple'>, msg=None), 'bearer_auth': Any(None, <class 'str'>, msg=None), 'opaque_id': Any(None, <class 'str'>, msg=None), 'headers': Any(None, <class 'dict'>, msg=None), 'connections_per_node': Any(None, All(Coerce(int, msg=None), Range(min=1, max=100, min_included=True, max_included=True, msg=None), msg=None), msg=None), 'http_compress': <function Boolean at 0x7f79e4134e00>, 'verify_certs': <function Boolean at 0x7f79e4135300>, 'ca_certs': Any(None, <class 'str'>, msg=None), 'client_cert': Any(None, <class 'str'>, msg=None), 'client_key': Any(None, <class 'str'>, msg=None), 'ssl_assert_hostname': Any(None, <class 'str'>, msg=None), 'ssl_assert_fingerprint': Any(None, <class 'str'>, msg=None), 'ssl_version': Any(None, <class 'str'>, msg=None), 'ssl_context': Any(None, <class 'str'>, msg=None), 'ssl_show_warn': <function Boolean at 0x7f79e41367a0>, 'transport_class': Any(None, <class 'str'>, msg=None), 'request_timeout': Any(None, All(Coerce(float, msg=None), Range(min=0.1, max=86400.0, min_included=True, max_included=True, msg=None), msg=None), msg=None), 'node_class': Any(None, <class 'str'>, msg=None), 'node_pool_class': Any(None, <class 'str'>, msg=None), 'randomize_nodes_in_pool': <function Boolean at 0x7f79e4136ac0>, 'node_selector_class': Any(None, <class 'str'>, msg=None), 'dead_node_backoff_factor': Any(None, <class 'float'>, msg=None), 'max_dead_node_backoff': Any(None, <class 'float'>, msg=None), 'serializer': Any(None, <class 'str'>, msg=None), 'serializers': Any(None, <class 'dict'>, msg=None), 'default_mimetype': Any(None, <class 'str'>, msg=None), 'max_retries': Any(None, All(Coerce(int, msg=None), Range(min=1, max=100, min_included=True, max_included=True, msg=None), msg=None), msg=None), 'retry_on_status': Any(None, <class 'tuple'>, msg=None), 'retry_on_timeout': <function Boolean at 0x7f79e4135bc0>, 'sniff_on_start': <function Boolean at 0x7f79e4136200>, 'sniff_before_requests': <function Boolean at 0x7f79e4148040>, 'sniff_on_node_failure': <function Boolean at 0x7f79e4148180>, 'sniff_timeout': Any(None, All(Coerce(float, msg=None), Range(min=0.1, max=100.0, min_included=True, max_included=True, msg=None), msg=None), msg=None), 'min_delay_between_sniffing': Any(None, All(Coerce(float, msg=None), Range(min=1, max=100.0, min_included=True, max_included=True, msg=None), msg=None), msg=None), 'sniffed_node_callback': Any(None, <class 'str'>, msg=None), 'meta_header': <function Boolean at 0x7f79e41484a0>, 'host_info_callback': Any(None, <class 'str'>, msg=None), '_transport': Any(None, <class 'str'>, msg=None)}}
2024-03-26 16:01:59,525 DEBUG es_client.helpers.schemacheck __init__:56 "Elasticsearch Configuration" config: {'client': {'hosts': 'https://infra-elasticsearch.infra:9200', 'http_compress': False, 'verify_certs': True}, 'other_settings': {'master_only': False, 'skip_version_test': False, 'username': 'elastic', 'password': 'REDACTED', 'api_key': {}}}
2024-03-26 16:02:00,114 CRITICAL es_client.helpers.config get_client:301 Unable to establish client connection to Elasticsearch!
2024-03-26 16:02:00,114 CRITICAL es_client.helpers.config get_client:302 Exception encountered: TLS error caused by: TlsError(TLS error caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)))
No worries. Found it. Will have an update out shortly.
@rgaduput, I apologize for the inconvenience. This is totally fixed now in Curator 8.0.13, which has been published to PyPI and Docker as well (as untergeek/curator:8.0.13).
Since you're using Kubernetes, you might be interested to know that you can avoid the config
YAML file altogether now using exclusively environment variables. You can see these at https://es-client.readthedocs.io/en/latest/envvars.html
@untergeek thanks for the quick check and the fix. sure will check the env vars config.
When upgraded the curator from 8.0.8 to 8.0.10 we have observed that it fails to connect with Elastic with TLS exception even though cert verify is set to
false
, this functionality works just fine if we go back to older version 8.0.8. please note i have tried the latest version 8.0.12 as well.Expected Behavior
Certs verification ignored while connecting to Elasticseach.
Actual Behavior
Fails to connect with below
Steps to Reproduce the Problem
Specifications
Context (Environment)
Trying to perform Elastic index clean up but ignore the certs while connecting to it since certs are self signed.
Detailed Description