AWS Elasticsearch keeps giving 403 unathorized

[alexanderfichel]

Here is my config.yaml

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: example_rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  minutes: 1

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 15

# The elasticsearch hostname for metadata writeback
# Note that every rule can have it's own elasticsearch host
es_host: search-itmcc-elasticsearch-cluster-47ux565rtlkza7gewdodwdh7b4.us-west-2.es.amazonaws.com

# The elasticsearch port
es_port: 80

aws_region: us-west-2

#boto_profile: default

# Optional URL prefix for elasticsearch
es_url_prefix: elasticsearch

# Connect with SSL to elasticsearch
#use_ssl: True

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

And the error that I get:

WARNING:elasticsearch:GET /elasticsearch/elastalert_status/elastalert/_search?size=1000 [status:403 request:0.015s]
WARNING:elasticsearch:GET /elasticsearch/elastalert_status/elastalert_status/_search?_source_include=endtime%2Crule_name&size=1 [status:403 request:0.005s]
ERROR:root:Error querying for last run: TransportError(403, u'{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet on resource: itmcc-elasticsearch-cluster"}')
WARNING:elasticsearch:POST /elasticsearch/elastalert_status/elastalert_error?op_type=create [status:403 request:0.003s]
ERROR:root:Error writing alert info to elasticsearch: TransportError(403, u'{"Message":"User: anonymous is not authorized to perform: es:ESHttpPost on resource: itmcc-elasticsearch-cluster"}')
Traceback (most recent call last):
  File "elastalert/elastalert.py", line 998, in writeback
    doc_type=doc_type, body=body)
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/__init__.py", line 248, in create
    return self.index(index, doc_type, body, id=id, params=params, op_type='create')
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/__init__.py", line 279, in index
    _make_path(index, doc_type, id), params=params, body=body)
  File "build/bdist.linux-x86_64/egg/elasticsearch/transport.py", line 329, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "build/bdist.linux-x86_64/egg/elasticsearch/connection/http_urllib3.py", line 109, in perform_request
    self._raise_error(response.status, raw_data)
  File "build/bdist.linux-x86_64/egg/elasticsearch/connection/base.py", line 108, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
AuthorizationException: TransportError(403, u'{"Message":"User: anonymous is not authorized to perform: es:ESHttpPost on resource: itmcc-elasticsearch-cluster"}')
WARNING:elasticsearch:GET /logstash-*/_search?_source_include=%40timestamp%2C%2A&ignore_unavailable=true&size=10000 [status:403 request:0.006s]
ERROR:root:Error running query: TransportError(403, u'{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet on resource: itmcc-elasticsearch-cluster"}')
WARNING:elasticsearch:POST /elasticsearch/elastalert_status/elastalert_error?op_type=create [status:403 request:0.004s]
ERROR:root:Error writing alert info to elasticsearch: TransportError(403, u'{"Message":"User: anonymous is not authorized to perform: es:ESHttpPost on resource: itmcc-elasticsearch-cluster"}')
Traceback (most recent call last):
  File "elastalert/elastalert.py", line 998, in writeback
    doc_type=doc_type, body=body)
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/__init__.py", line 248, in create
    return self.index(index, doc_type, body, id=id, params=params, op_type='create')
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/__init__.py", line 279, in index
    _make_path(index, doc_type, id), params=params, body=body)
  File "build/bdist.linux-x86_64/egg/elasticsearch/transport.py", line 329, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "build/bdist.linux-x86_64/egg/elasticsearch/connection/http_urllib3.py", line 109, in perform_request
    self._raise_error(response.status, raw_data)
  File "build/bdist.linux-x86_64/egg/elasticsearch/connection/base.py", line 108, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
AuthorizationException: TransportError(403, u'{"Message":"User: anonymous is not authorized to perform: es:ESHttpPost on resource: itmcc-elasticsearch-cluster"}')

My IAM role attached to instance with elastalert gives full access to es (elasticsearch):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1454694581000",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1454694609000",
            "Effect": "Allow",
            "Action": [
                "events:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1454694630000",
            "Effect": "Allow",
            "Action": [
                "logs:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1454694649000",
            "Effect": "Allow",
            "Action": [
                "es:*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

I confirmed with IAM policy simulator as well as other cloudformation instances that use other apps to access elasticsearch in AWS:

[Qmando]

Hmm.. you do have aws_region set, which means it should be signing the requests using the instance role. I'm not entirely sure why that's not working. I think adding --es_debug_trace file will let you see if it's actually signing the requests or not.

You could try using a boto profile instead. This will make it look for credentials in ~/.aws/credentials. See http://elastalert.readthedocs.io/en/latest/recipes/signing_requests.html?highlight=aws.

Unrelated, but you probably don't need the es_url_prefix with amazon's ES.

[alexanderfichel]

It is like it is not picking the aws_region: us-west-2. If I starting to put in mistakes in the key name or value name, nothing changes, as if it is being ignored. Tried with boto profile too. When I try to run the second command from first install docs ():

$ python setup.py install (runs okay)
$ pip install -r requirements.txt

Second command gives error:

    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DBLIST_FLOAT_RADIX_SORT=1 -I/usr/include/python2.7 -c blist/_blist.c -o build/temp.linux-x86_64-2.7/blist/_blist.o
    blist/_blist.c:38:20: fatal error: Python.h: No such file or directory
     #include <Python.h>
                        ^
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
  Rolling back uninstall of blist
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-cPRJIB/blist/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-BGxTKy-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-cPRJIB/blist/
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
You are using pip version 8.1.0, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[Qmando]

Oh, just realized that you need to put aws_region and boto_profile in the RULE yaml and not config.yaml.

Also, I think you need to run apt-get install python-devel to fix the Python.h missing error. But, if setup.py install works fine, you should have all the necessary libraries.

[alexanderfichel]

I tried putting aws_region in the rule, and still no luck. --es_debug_trace file create empty file even with --debug and --verbose command parameters

[Qmando]

You could put put a print statement here, https://github.com/Yelp/elastalert/blob/master/elastalert/auth.py#L42, to see if the credentials are even getting fetched properly. Like print aws_access_key_id, aws_token

Beyond trying boto_profile, I'm not sure what else to do.

[alexanderfichel]

I printed out the aws parameters in the auth file

print '---------------------------------------------------' print aws_access_key_id print aws_secret_access_key print aws_token print host print aws_region print '---------------------------------------------------'

but I still get error:

Something about "AttributeError: 'AWSRequestsAuth' object has no attribute 'split'"

---------------------------------------------------
[ACCESS_KEY]
[SECRET_KEY]
[TOKEN_HIDDEN]
search-itmcc-elasticsearch-cluster-47ux565rtlkza7gewdodwdh7b4.us-west-2.es.amazonaws.com
us-west-2
---------------------------------------------------
ERROR:root:Traceback (most recent call last):
  File "elastalert/elastalert.py", line 753, in run_all_rules
    num_matches = self.run_rule(rule, endtime, self.starttime)
  File "elastalert/elastalert.py", line 516, in run_rule
    self.current_es = self.new_elasticsearch(rule_es_conn_config)
  File "elastalert/elastalert.py", line 143, in new_elasticsearch
    send_get_body_as=es_conn_conf['send_get_body_as'])
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 150, in __init__
    self.transport = transport_class(_normalize_hosts(hosts), **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 113, in __init__
    self.set_connections(hosts)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 166, in set_connections
    connections = map(_create_connection, hosts)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 165, in _create_connection
    return self.connection_class(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_requests.py", line 35, in __init__
    http_auth = http_auth.split(':', 1)
AttributeError: 'AWSRequestsAuth' object has no attribute 'split'

ERROR:root:Uncaught exception running rule Example rule: 'AWSRequestsAuth' object has no attribute 'split'
WARNING:elasticsearch:POST http://search-itmcc-elasticsearch-cluster-47ux565rtlkza7gewdodwdh7b4.us-west-2.es.amazonaws.com:80/elastalert_status/elastalert_error?op_type=create [status:403 request:0.007s]
ERROR:root:Error writing alert info to elasticsearch: TransportError(403, u'{"Message":"User: anonymous is not authorized to perform: es:ESHttpPost on resource: itmcc-elasticsearch-cluster"}')
Traceback (most recent call last):
  File "elastalert/elastalert.py", line 1022, in writeback
    doc_type=doc_type, body=body)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 68, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 227, in create
    return self.index(index, doc_type, body, id=id, params=params, op_type='create')
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 68, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 257, in index
    _make_path(index, doc_type, id), params=params, body=body)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 301, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_requests.py", line 72, in perform_request
    self._raise_error(response.status_code, raw_data)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py", line 102, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
AuthorizationException: TransportError(403, u'{"Message":"User: anonymous is not authorized to perform: es:ESHttpPost on resource: itmcc-elasticsearch-cluster"}')

[Qmando]

https://github.com/DavidMuller/aws-requests-auth/issues/15#issuecomment-199323512

Try upgrading the elasticsearch library to 1.7?

My bad if the library versions are incompatible, I've never tested this feature myself.

[alexanderfichel]

No luck with that either. I decided to start from scratch and change the package to 1.7 in requirement.txt, but I run into this error when i run python setup.py install:

Installed /tmp/easy_install-MCuRSg/mock-2.0.0/.eggs/pbr-1.10.0-py2.7.egg
Marker evaluation failed, see the following error.  For more information see: http://docs.openstack.org/developer/pbr/compatibility.html#evaluate-marker
ERROR:root:Error parsing
Traceback (most recent call last):
  File "/tmp/easy_install-MCuRSg/mock-2.0.0/.eggs/pbr-1.10.0-py2.7.egg/pbr/core.py", line 111, in pbr
    attrs = util.cfg_to_args(path, dist.script_args)
  File "/tmp/easy_install-MCuRSg/mock-2.0.0/.eggs/pbr-1.10.0-py2.7.egg/pbr/util.py", line 248, in cfg_to_args
    kwargs = setup_cfg_to_setup_kwargs(config, script_args)
  File "/tmp/easy_install-MCuRSg/mock-2.0.0/.eggs/pbr-1.10.0-py2.7.egg/pbr/util.py", line 431, in setup_cfg_to_setup_kwargs
    if pkg_resources.evaluate_marker('(%s)' % env_marker):
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1483, in evaluate_marker
    return cls.interpret(parser.expr(text).totuple(1)[1])
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1517, in interpret
    return op(nodelist)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1431, in atom
    return cls.interpret(nodelist[2])
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1517, in interpret
    return op(nodelist)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1452, in comparison
    raise SyntaxError(msg)
SyntaxError: '<' operator not allowed in environment markers
error: Setup script exited with error in setup command: Error parsing /tmp/easy_install-MCuRSg/mock-2.0.0/setup.cfg: SyntaxError: '<' operator not allowed in environment markers

[alexanderfichel]

Would love to gt this working, what libraries do you currently use in the requirements txt as well as OS, are you using any instance in AWS? Is it ubuntu/rhel/amazon linux etc?

[Qmando]

Someone else had this problem, https://github.com/marcan/letsencrypt-external/issues/1#issuecomment-212412568

Maybe upgrading setuptools will fix this last error. You'll need all the libraries otherwise there will be import errors. The versions are not required to all be exact though.

We are using this on Ubuntu.

[Qmando]

You could also try using Docker.

https://github.com/krizsan/elastalert-docker or https://github.com/fiunchinho/docker-elastalert or https://www.ivankrizsan.se/2015/10/19/creating-an-elastalert-docker-image-on-docker-hub/

Or if not, definitely use a virtualenv. http://docs.python-guide.org/en/latest/dev/virtualenvs/ and try installing with pip install -r requirements.txt rather than setup.py install.

[alexanderfichel]

Finally got it working, thanks a lot Qmando! I haven't used the docker, but had to update some libraries when I built it on the amazon linux instace:

pip install -U setuptools
pip install -U mock
pip install -U pbr
pip install -U requests-oauthlib

As well as update in requirements.txt

configparser==3.5.0
elasticsearch==1.7.0

[abcondas]

I ran into this problem when attempting to run a bucketing search in es (also despite having the "es:*" action on my es domain resource). I was able to resolve it by adding the following to my lambda role. Hope it gives you some ideas...

• Effect: Allow Action:

  • es:ESHttpGet Resource:
  • !Join [":", ["arn:aws:es", !Ref "AWS::Region", !Ref "AWS::AccountId", "domain//*/_search"]]