Can't be used with Amazon Elasticsearch service

[exidy]

This connector can't be used in conjunction with the Amazon Elasticsearch service because it requires the ES transport protocol, which Amazon ES doesn't expose. Would it be possible for this connector to use the REST protocol?

[Jinkxed]

Would love to see this as well. No point in standing up your own ES clusters anymore :)

[dvassallo]

Hello - We're looking into making this consumer compatible with the Amazon Elasticsearch Service. In the meantime, CloudWatch Logs offers an alternative integration option with the Amazon Elasticsearch Service which you can set up from the AWS Management Console. If you already have an Amazon ES cluster set up, you can simply click on a log group from the CloudWatch Logs section and choose "Actions -> Start Streaming to Amazon Elasticsearch Service":

That solution uses a Lambda function to convert the CWL logs to ES documents and it is implemented very similarly to this consumer application. One other benefit of that setup is that you do not need to run any EC2 instances either. It's a completely serverless setup between CWL and your ES cluster.

[exidy]

Unfortunately this connector can't be used in AWS Regions without Lambda (e.g. Sydney)

[dvassallo]

Yes unfortunately the feature in the AWS Management Console is only available in AWS regions where Lambda is currently available. However it is actually possible to have just the Lambda function in one region (e.g. Toyko) and have the Amazon ES cluster and the CWL log group in another (e.g Sydney).

The following solution is definitely a sub-optimal setup experience, but it's just a one-time effort and should be quite straight forward:

• Create an empty CWL log group and a small Amazon ES cluster in Tokyo (both are temporary).
• Connect the two together with the "Stream to Amazon Elasticsearch Service" wizard.
• Once you get that done, you can delete your empty CWL log group and the Amazon ES cluster in Tokyo.
• Find the Lambda function in the Tokyo region. It should have a name with the following format: LogsToElasticsearch_<AES_DOMAIN_NAME>.
• Click on the "Code" tab and edit line 6. Put the Amazon ES endpoint from your Sydney cluster there. You should be able to obtain the endpoint from the AWS Console. Save the Lambda function.
• The only thing that would remain is to connect your CWL log group(s) in Sydney to the Tokyo Lambda function (which would be pointing to your Syndey Amazon ES cluster). Unfortunately this step cannot be done from the AWS Console yet so you may want to do it with the AWS CLI:

aws logs put-subscription-filter \
   --log-group-name "<LOG_GROUP_NAME>" \
   --filter-name "LambdaToElasticsearch" \
   --filter-pattern "" \
   --destination-arn "arn:aws:lambda:ap-southeast-2:<AWS_ACCOUNT_ID>:function:<LAMBDA_FUNCTION_NAME>"

The filter pattern option is important to have the fields properly indexed in Elasticsearch (unless your log data is in JSON format). You may want to check the Getting CloudWatch Logs data indexed in Elasticsearch section in the README.md of this project for more info and example filter patterns for common log formats.

[NareshDealer]

@DVassallo Its very interesting alternative way to use. Thank you for sharing this info.

However, i have one tricky situation where i want to get some guidance.

I have a setup where i streamed all log groups to elastic search cluster service using lambda. Now that we achieved that part, we want to setup our inhouse Elastic search cluster( for various reasons ) where i want to use the lambda function to send those cloudwatch logs to our inhouse cluster. I thought of modifying that lambda function to point it to our cluster but it doesnt seems to be liking that.

Any guidance or help is really appreciated.

Thanks.

[vegardvaage]

@DVassallo is the above approach still the best alternative for this? I'm trying to solve a cross-account CloudTrail -> centralized AWS ES setup in the most streamlined manner possible.

[Nomane]

It is also possible to convert this trick (CW logs to AES) in Cloudformation as well ?

Thanks