sshivanii
commented
2 years ago Please see below for common functionality and Plugin differences between the logstash-output-amazon_es and logtstash-output-opensearch plugins.
Common Functionality
- user
- password
- ssl
- Port: Default is 443 for both Amazon Elasticsearch Service and OpenSearch
- Batch size: 20MB. If single document exceeds 20MB it is sent as a single request.
- Errors retried infinitely: Network Errors, 429, 503
- Valid APIs: index, delete, create, update, sprintf
- Default service name: es
Plugin Differences
| logstash-output-amazon_es | logstash-output-opensearch | |
|---|---|---|
| Plugin name | amazon_es | opensearch |
| Plugin support | Stores logs in Elasticsearch, compatible with Kibana | Stores logs in OpenSearch, compatible with OpenSearch Dashboards |
| Auth Type | AWS IAM | AWS IAM, extensible to custom auth_type |
| Default index value | logstash-%{+YYYY.MM.dd} | ecs_compatibility disabled: logstash-%{+yyyy.MM.dd} ecs_compatibility enabled: ecs-logstash-%{+yyyy.MM.dd} |
| ECS compatibility | Not Compatible | Compatible |
| Example config | output { amazon_es { hosts => ["foo.us-east-1.es.amazonaws.com"] region => "us-east-1" # aws_access_key_id and aws_secret_access_key are optional if instance profile is configured aws_access_key_id => 'ACCESS_KEY' aws_secret_access_key => 'SECRET_KEY' index => "production-logs-%{+YYYY.MM.dd}" }} | output { opensearch { hosts => ["hostname:port"] auth_type => { type => 'aws_iam' aws_access_key_id => 'ACCESS_KEY' aws_secret_access_key => 'SECRET_KEY' region => 'us-west-2' } index => "logstash-logs-%{+YYYY.MM.dd}" } } |
The OpenSearch project has created a new Logstash output plugin which can send events to OpenSearch 1.x or Elasticsearch 7.x clusters - logstash-output-opensearch. Starting with logstash-output-opensearch 1.2 it also supports SigV4 signing.
Having both the logstash-output-opensearch and logstash-output-amazon_es plugins can be confusing since it becomes unclear which one to use. With that in mind, we are looking into eventually replacing the logstash-output-amazon_es plugin with the logstash-output-opensearch plugin.
For now, we will place the logstash-output-amazon_es plugin into a maintenance mode. We will supply critical bug fixes and security patches. But, new features and functionality should be considered for the logstash-output-opensearch plugin instead.
Both plugins have some overlapping compatibility:
As noted in the above table, the logstash-output-opensearch plugin does not support Elasticsearch 6.x. I've opened a feature request on that repo to support it - https://github.com/opensearch-project/logstash-output-opensearch/issues/123.
We will provide a migration guide to help teams and individuals migrate to the logstash-output-opensearch plugin.
We will also triage issues from this GitHub repository and add them to the logstash-output-opensearch repository as needed. We will mostly focus on feature requests.