Trying to parse RDS logs

[ShaneCal]

Hi,

I am outputting my RDS logs to cloudwatch logs and am trying to pass them to ES using this plugin.

My configuration file looks like this:

input{
  cloudwatch {
    access_key_id => "xxx"
    secret_access_key => "xxx"
    log_group => ["/aws/rds/cluster/rdscluster01/postgresql"]
    region => "us-east-1"
    add_field => {
        "doctype" => "aws-cloudwatch-logs"
        "es_index" => "cloudwatch-logs"
        }
    }
}

output {
        elasticsearch {
            hosts => [ "https://myelasticsearchcluster.us-east-1.es.amazonaws.com:443" ]
            index => "%%{[es_index]}-%%{+YYYY.MM.dd}"
        }
}

Here are a couple of examples of the messages from the RDS logs that I want to be returned to ES:

2020-03-04 12:13:46 UTC:10.1.47.201(31531):[unknown]@[unknown]:[3632]:LOG:  connection received: host=10.1.xx.xx port=31531
2020-03-04 12:13:46 UTC:10.1.47.201(31531):user@database:[3632]:LOG:  connection authorized: user=user database=database

I am a little confused here because there are no traces of these messages in what is being returned to ES right now. Here is the JSON currently being returned to ES:

{
  "_index": "cloudwatch-logs-2020.03.04",
  "_type": "doc",
  "_id": "Ah8nq3ABnLhsi5972BFJ",
  "_version": 1,
  "_score": null,
  "_source": {
    "tags": [
      "_grokparsefailure",
      "_dateparsefailure",
      "_geoip_lookup_failure"
    ],
    "cloudwatch_logs": {
      "ingestion_time": "2020-03-04T22:16:21.362Z",
      "log_group": "/aws/rds/cluster/rdscluster01/postgresql",
      "log_stream": "rdscluster01.0",
      "event_id": "35310111909378837996277702224072479283437742765700743171"
    },
    "es_index": "cloudwatch-logs",
    "@timestamp": "2020-03-04T22:16:19.000Z",
    "@version": "1",
    "doctype": "aws-cloudwatch-logs"
  },
  "fields": {
    "cloudwatch_logs.ingestion_time": [
      "2020-03-04T22:16:21.362Z"
    ],
    "@timestamp": [
      "2020-03-04T22:16:19.000Z"
    ]
  },
  "sort": [
    1583360179000
  ]
}

Can someone help me getting these messages back and parsing them according to field?

[ShaneCal]

@lukewaite, am I missing something obvious to get this going?

[srolskyi]

@lukewaite please help

[alona-shevliakova]

@ShaneCal seems to me if you use logstash-input-cloudwatch-logs plugin your configuration has to look like mentioned here https://github.com/lukewaite/logstash-input-cloudwatch-logs#example input { cloudwatch_logs { instead provided input{ cloudwatch { It may be just a typo.