search

logstash-plugins / logstash-output-elastic_app_search

Apache License 2.0
0 stars 9 forks source link

Convert datetime with offset to RFC3339 before sending to App Search #10

Open ppf2 opened 5 years ago

ppf2 commented 5 years ago

Have a Logstash ES input fetching data from an ES index running on 6.8.1.

The index has a field defined below:

"CreatedDate" : {
            "type" : "date"
          }

With source values that contain a timezone offset: "CreatedDate" : "2019-08-05T04:07:00.000+0000"

Feeding the above into the elastic_app_search plugin results in the event getting dropped:

        "errors": [
            "Invalid field value: Value '2019-08-05T04:07:00.000+0000' cannot be parsed as a date (RFC 3339)"
        ]

It does not support +0000, but +00:00 works.

I am working around this by converting the ISO8601 timestamp to RFC3339 in a custom ruby filter: DateTime.strptime(event.get('CreatedDate'), '%Y-%m-%dT%H:%M:%S.%L%z').rfc3339.to_s

It will be nice if the logstash-output-elastic_app_search plugin can handle the conversion to RFC3339 format automatically.

jsvd commented 5 years ago

"CreatedDate" : "2019-08-05T04:07:00.000+0000"

who is producing these timestamps? date format conciliation is typically handled in the date filter, I'm a bit concerned with adding this functionality to this output plugin

ppf2 commented 5 years ago

These are coming from the source data and Elasticsearch accepted it as the _source value of the date field. I have opened an issue with Swiftype team to discuss if it makes sense for the App Search service to accept both ISO8601 and RFC3339 formats for consistency across products. If we can address it there, we will not have to do this in the output plugin. thx!

billsinc commented 4 years ago

@ppf2 hopefully so. I'm having issues with App Search not accepting dates from Postgres through Logstash either.