How to use ESS command with ES and shield implemnted

praveer19 commented 5 years ago

Basically, i want to search against the ES cluster where shield plugin is implemented and i need to provide authentication details..With the current implementation of this app, i m getting the following error:

External search command 'ess' returned error code 1. Script output = "error_message=AuthenticationException at "/Users/jigsaw/Documents/splunk/etc/apps/elasticsplunk/bin/elasticsearch/connection/base.py", line 125 : TransportError(401, u'security_exception', u'missing authentication token for REST request [/quotingreport*/_search?size=10000&scroll=5m]') "

Please help how to solve this...

brunotm commented 5 years ago

Hi @praveer19 Unfortunately i don't have a shield setup to test, but i understand it supports basic auth.

Can you try specifying your eaddr like: |ess eaddr="https://USER:PASS@node1:9200,https://USER:PASS@node2:9200"

And post your results ?

Cheers!

ecwhipple commented 5 years ago

Hi, Having a similar experience connecting Splunk 7.1.3 to Elasticsearch 6.6.0 secured with SearchGuard plugin. I am trying to connect a couple different ways: |ess eaddr=https://node1:9200 produces the following error: External search command 'ess' returned error code 1. Script output = "error_message=ConnectionError at "/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsearch/connection/http_urllib3.py", line 156 : ConnectionError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f9e1162ec10>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f9e1162ec10>: Failed to establish a new connection: [Errno 111] Connection refused) "

And the following log output from splunkd.log: 03-07-2019 15:53:05.336 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,335, Level=DEBUG, Pid=11434, Logger=splunklib, File=search_command.py, Line=572, ElasticSplunk.process started under protocol_version=1 03-07-2019 15:53:05.336 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,336, Level=DEBUG, Pid=11434, Logger=splunklib, File=search_command.py, Line=579, Writing configuration settings 03-07-2019 15:53:05.337 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,337, Level=DEBUG, Pid=11434, Logger=splunklib, File=search_command.py, Line=508, metadata={u'action': u'getinfo', u'preview': True, u'searchinfo': {u'session_key': None, u'app': None, u'raw_args': ['/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py', '__GETINFO__', 'eaddr=https://192.168.33.50:9200'], u'splunkd_uri': None, u'owner': None, u'args': ['/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py', '__GETINFO__', 'eaddr=https://192.168.33.50:9200'], u'earliest_time': None, u'dispatch_dir': None, u'username': None, u'search': u'|ess eaddr=https://192.168.33.50:9200', u'latest_time': None, u'sid': u'', u'splunk_version': u'7.1.3'}}, input_header={u'keywords': u'""', u'preview': u'0', u'search': u'|ess eaddr=https://192.168.33.50:9200', u'sid': u'', u'truncated': u'0', u'allowStream': u'1', u'splunkVersion': u'7.1.3', u'realtime': u'0'} 03-07-2019 15:53:05.337 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,337, Level=DEBUG, Pid=11434, Logger=splunklib, File=search_command.py, Line=515, tempfile.tempdir=None 03-07-2019 15:53:05.337 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,337, Level=DEBUG, Pid=11434, Logger=splunklib, File=internals.py, Line=119, Parsing ElasticSplunk command line: ['eaddr=https://192.168.33.50:9200'] 03-07-2019 15:53:05.337 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,337, Level=DEBUG, Pid=11434, Logger=splunklib, File=internals.py, Line=155, ElasticSplunk: elasticsplunk eaddr="https://192.168.33.50:9200" 03-07-2019 15:53:05.345 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,338, Level=DEBUG, Pid=11434, Logger=splunklib, File=search_command.py, Line=624, ElasticSplunk.process finished under protocol_version=1 03-07-2019 15:53:05.918 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,913, Level=DEBUG, Pid=11464, Logger=splunklib, File=search_command.py, Line=572, ElasticSplunk.process started under protocol_version=1 03-07-2019 15:53:05.919 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,913, Level=DEBUG, Pid=11464, Logger=splunklib, File=search_command.py, Line=579, Writing configuration settings 03-07-2019 15:53:05.919 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,913, Level=DEBUG, Pid=11464, Logger=splunklib, File=search_command.py, Line=508, metadata={u'searchinfo': {u'dispatch_dir': None, u'splunkd_uri': None, u'username': None, u'earliest_time': None, u'sid': u'searchparsetmp_820669171', u'app': None, u'args': ['/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py', '__GETINFO__', 'eaddr=https://192.168.33.50:9200'], u'raw_args': ['/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py', '__GETINFO__', 'eaddr=https://192.168.33.50:9200'], u'splunk_version': u'7.1.3', u'owner': None, u'search': u'|ess eaddr=https://192.168.33.50:9200', u'session_key': None, u'latest_time': None}, u'preview': True, u'action': u'getinfo'}, input_header={u'realtime': u'0', u'keywords': u'""', u'truncated': u'0', u'preview': u'0', u'search': u'|ess eaddr=https://192.168.33.50:9200', u'allowStream': u'1', u'sid': u'searchparsetmp_820669171', u'splunkVersion': u'7.1.3'} 03-07-2019 15:53:05.919 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,914, Level=DEBUG, Pid=11464, Logger=splunklib, File=search_command.py, Line=515, tempfile.tempdir=None 03-07-2019 15:53:05.919 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,914, Level=DEBUG, Pid=11464, Logger=splunklib, File=internals.py, Line=119, Parsing ElasticSplunk command line: ['eaddr=https://192.168.33.50:9200'] 03-07-2019 15:53:05.919 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,914, Level=DEBUG, Pid=11464, Logger=splunklib, File=internals.py, Line=155, ElasticSplunk: elasticsplunk eaddr="https://192.168.33.50:9200" 03-07-2019 15:53:05.919 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,914, Level=DEBUG, Pid=11464, Logger=splunklib, File=search_command.py, Line=624, ElasticSplunk.process finished under protocol_version=1

When I execute with the 'admin:pass' in the search, I get the same error in the Splunk UI: |ess eaddr=https://admin:<pass>@192.168.33.50:9200

UI error: External search command 'ess' returned error code 1. Script output = "error_message=ConnectionError at "/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsearch/connection/http_urllib3.py", line 156 : ConnectionError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6f4234ec50>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6f4234ec50>: Failed to establish a new connection: [Errno 111] Connection refused) "

And the following log set from splunkd.log: 03-07-2019 15:54:12.727 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:12,719, Level=DEBUG, Pid=11601, Logger=splunklib, File=search_command.py, Line=572, ElasticSplunk.process started under protocol_version=1 03-07-2019 15:54:12.727 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:12,719, Level=DEBUG, Pid=11601, Logger=splunklib, File=search_command.py, Line=579, Writing configuration settings 03-07-2019 15:54:12.727 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:12,719, Level=DEBUG, Pid=11601, Logger=splunklib, File=search_command.py, Line=508, metadata={u'action': u'getinfo', u'preview': True, u'searchinfo': {u'session_key': None, u'search': u'|ess eaddr=https://admin:<pass>@192.168.33.50:9200', u'owner': None, u'sid': u'', u'earliest_time': None, u'username': None, u'splunk_version': u'7.1.3', u'args': ['/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py', '__GETINFO__', 'eaddr=https://admin:<pass>@192.168.33.50:9200'], u'splunkd_uri': None, u'dispatch_dir': None, u'raw_args': ['/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py', '__GETINFO__', 'eaddr=https://admin:<pass>@192.168.33.50:9200'], u'app': None, u'latest_time': None}}, input_header={u'keywords': u'""', u'allowStream': u'1', u'realtime': u'0', u'preview': u'0', u'sid': u'', u'splunkVersion': u'7.1.3', u'search': u'|ess eaddr=https://admin:<pass>@192.168.33.50:9200', u'truncated': u'0'} 03-07-2019 15:54:12.727 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:12,719, Level=DEBUG, Pid=11601, Logger=splunklib, File=search_command.py, Line=515, tempfile.tempdir=None 03-07-2019 15:54:12.727 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:12,719, Level=DEBUG, Pid=11601, Logger=splunklib, File=internals.py, Line=119, Parsing ElasticSplunk command line: ['eaddr=https://admin:<pass>@192.168.33.50:9200'] 03-07-2019 15:54:12.727 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:12,720, Level=DEBUG, Pid=11601, Logger=splunklib, File=internals.py, Line=155, ElasticSplunk: elasticsplunk eaddr="https://admin:<pass>@192.168.33.50:9200" 03-07-2019 15:54:12.727 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:12,720, Level=DEBUG, Pid=11601, Logger=splunklib, File=search_command.py, Line=624, ElasticSplunk.process finished under protocol_version=1 03-07-2019 15:54:13.283 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:13,282, Level=DEBUG, Pid=11632, Logger=splunklib, File=search_command.py, Line=572, ElasticSplunk.process started under protocol_version=1 03-07-2019 15:54:13.283 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:13,282, Level=DEBUG, Pid=11632, Logger=splunklib, File=search_command.py, Line=579, Writing configuration settings 03-07-2019 15:54:13.283 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:13,283, Level=DEBUG, Pid=11632, Logger=splunklib, File=search_command.py, Line=508, metadata={u'preview': True, u'searchinfo': {u'sid': u'searchparsetmp_1008607785', u'splunk_version': u'7.1.3', u'latest_time': None, u'search': u'|ess eaddr=https://admin:<pass>@192.168.33.50:9200', u'username': None, u'dispatch_dir': None, u'earliest_time': None, u'args': ['/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py', '__GETINFO__', 'eaddr=https://admin:<pass>@192.168.33.50:9200'], u'owner': None, u'raw_args': ['/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py', '__GETINFO__', 'eaddr=https://admin:<pass>@192.168.33.50:9200'], u'session_key': None, u'splunkd_uri': None, u'app': None}, u'action': u'getinfo'}, input_header={u'sid': u'searchparsetmp_1008607785', u'realtime': u'0', u'splunkVersion': u'7.1.3', u'search': u'|ess eaddr=https://admin:<pass>@192.168.33.50:9200', u'allowStream': u'1', u'preview': u'0', u'truncated': u'0', u'keywords': u'""'} 03-07-2019 15:54:13.283 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:13,283, Level=DEBUG, Pid=11632, Logger=splunklib, File=search_command.py, Line=515, tempfile.tempdir=None 03-07-2019 15:54:13.283 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:13,283, Level=DEBUG, Pid=11632, Logger=splunklib, File=internals.py, Line=119, Parsing ElasticSplunk command line: ['eaddr=https://admin:<pass>@192.168.33.50:9200'] 03-07-2019 15:54:13.283 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:13,283, Level=DEBUG, Pid=11632, Logger=splunklib, File=internals.py, Line=155, ElasticSplunk: elasticsplunk eaddr="https://admin:<pass>@192.168.33.50:9200" 03-07-2019 15:54:13.284 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:13,283, Level=DEBUG, Pid=11632, Logger=splunklib, File=search_command.py, Line=624, ElasticSplunk.process finished under protocol_version=1

I've tried to edit the Urllib3HttpConnection(Connection) class in /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsearch/connection/http_urllib3.py to provide different values and those error as well.

Help is appreciated, Thanks!

brunotm commented 5 years ago

@ecwhipple, the error from the logs is 111 Connection refused. Is https://192.168.33.50:9200 the actual elasticsearch url? do you have ssl enabled for elasticsearch itself (not kibana...)? If so elasticsearch is listening on port 9200?

Can you post your elasticsearch.url from your kibana.yml configuration file or from the kibana command line if specified there ?

ecwhipple commented 5 years ago

@brunotm I did some netcat tests from the splunk instance (test-centos-7) to the elasticsearch instance (test-single-centos-7) and found I was getting connection refused messages. [root@test-centos-7 ~]# nc -vz 192.168.33.50 9200 Ncat: Version 7.50 ( https://nmap.org/ncat ) Ncat: Connection refused.

I updated the elasticsearch.yml, it was listening on 127.0.0.1 and is now updated to host: 0.0.0.0 and can verify that I am able to curl that elasticsearch endpoint from the splunk instance: [root@test-centos-7 ~]# curl -k https://$CREDS@192.168.33.50:9200 { "name" : "test-single-centos-7", "cluster_name" : "elk_test_cluster", "cluster_uuid" : "mIFQ8PaxT7W39D4LnyoaNw", "version" : { "number" : "6.6.0", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "a9861f4", "build_date" : "2019-01-24T11:27:09.439740Z", "build_snapshot" : false, "lucene_version" : "7.6.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }

Now, when running the following splunk search, I am getting a new error: Search: |ess eaddr=$CREDS@192.168.33.50:9200 action=cluster-health Error: External search command 'ess' returned error code 1. Script output = "error_message=ConnectionError at "/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsearch/connection/http_urllib3.py", line 156 : ConnectionError(('Connection aborted.', BadStatusLine("''",))) caused by: ProtocolError(('Connection aborted.', BadStatusLine("''",))) "

Is that due to the ssl not being passed in the search?

brunotm commented 5 years ago

@ecwhipple the eaddr parameter must specify the protocol, otherwise will default to http. So your eaddr would be https://$CREDS@192.168.33.50:9200. Additionally you can create a cluster configuration in the $SPLUNK_PATH//etc/apps/elasticsplunk/local/elasticsplunk.json:

{
    "cluster1":{
        "hosts": ["https://$CREDS@192.168.33.50:9200"],
        "tsfield": "timestamp",
        "verify_certs": false
    },

    "cluster2":{
        "hosts": ["https://node1/elastic", "https://node2/elastic", "https://node3/elastic"],
        "tsfield": "time",
        "verify_certs": true,
                 "ca_cert": "/path/to/cert/ca.pem"
    }
}

and reference the named cluster in eaddr, e.g. |ess eaddr=cluster1

ecwhipple commented 5 years ago

Thanks @brunotm, this is working now. Appreciate the help and fast response.

brunotm / elasticsplunk

How to use ESS command with ES and shield implemnted #22