[BUG] RequestResponseSerializer DateTime serialization is inconsistent

[cspital]

What is the bug?

DateRange queries of the form: Query(q => q.Bool(b => b.Must(m => m.DateRange(...)))) produce inconsistently serialized DateTime values and server errors. RequestResponseSerializer appears to produce unparseable DateTime strings in some use cases.

How can one reproduce the bug?

public class FacilityModelV2
{
    ... snipped ...

    [Keyword(Index = true)]
    public string Key { get; set; }

    // stored in format: yyyy'-'MM'-'dd'T'HH':'mm':'ss'.'fffzzz
    [Date(Format = "date_time")]
    public DateTime RepositoryTimeStamp { get; set; }
}

var results = await _client.SearchAsync<FacilityModelV2>(s => s
    .Index("some_index")
    .TrackTotalHits(true)
    .Size(1000)
    .From(0)
    .Sort(x => x.Ascending(z => z.Key))
    .Query(q => q.Bool(b => b.Must(m => m.DateRange(r => r.Field(f => f.RepositoryTimeStamp).GreaterThanOrEquals(new DateTime(2022, 11, 18, 11, 6, 55, 0, DateTimeKind.Local))))))
    );

// Additionally, one can observe the incorrect serialization behavior without talking to the server at all.
Console.WriteLine(_client.RequestResponseSerializer.SerializeToString(new DateTime(2022, 11, 18, 11, 6, 55, 0, DateTimeKind.Local)));
Console.WriteLine(_client.RequestResponseSerializer.SerializeToString(new DateTime(2022, 11, 18, 11, 6, 55, 1, DateTimeKind.Local)));

Outputs the datetime value as 2022-11-18T11:06:55-08:00 and the call fails with the following exception: OpenSearch.Net.OpenSearchClientException: Request failed to execute. Call: Status code 400 from: POST /some_index/_search?typed_keys=true. ServerError: Type: search_phase_execution_exception Reason: "all shards failed" Stack Trace:  Transport1.HandleOpenSearchClientException(RequestData data, Exception clientException, IOpenSearchResponse response) line 253 Transport1.FinalizeResponse[TResponse](RequestData requestData, IRequestPipeline pipeline, List1 seenExceptions, TResponse response) line 221 Transport1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters) OpenSearchClient.SearchAsync[TDocument](ISearchRequest request, CancellationToken ct) line 895 OpenSearchClient.SearchAsync[TDocument](Func`2 selector, CancellationToken ct) line 881

If we set the milliseconds to a non-default value like 123, the value is output correctly as 2022-11-18T11:06:55.123-08:00 and the query succeeds. If, using Kibana, we force the DateTime to look like: 2022-11-18T11:06:55.0-08:00 it also works. So it appears the issue is in the RequestResponseSerializer's handling of DateTime values.

What is the expected behavior?

The RequestResponseSerializer should always output values in a format that the server can parse and understand, a DateTime with a 0 millisecond value is valid and expected in the real world.

What is your host/environment?

AWS Hosted OpenSearch (2.3)/OpenSearch.Client (1.2)/Windows 10

Do you have any screenshots?

No.

Do you have any additional context?

Even forcing a value like 2022-11-18T11:06:55.-08:00 through Kibana results in a successful query. The query in Kibana used to test this as follows:

GET some_index/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "RepositoryTimeStamp": {
              "gte": "2022-11-18T11:06:55-08:00"
            }
          }
        }
      ]
    }
  }
}

This fails with the following server error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "failed to parse date field [2022-11-18T11:06:55-08:00] with format [date_time]: [failed to parse date field [2022-11-18T11:06:55-08:00] with format [date_time]]"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "some_index",
        "node" : "kP_Qc_DJRE6jlrMzSacNlw",
        "reason" : {
          "type" : "parse_exception",
          "reason" : "failed to parse date field [2022-11-18T11:06:55-08:00] with format [date_time]: [failed to parse date field [2022-11-18T11:06:55-08:00] with format [date_time]]",
          "caused_by" : {
            "type" : "illegal_argument_exception",
            "reason" : "failed to parse date field [2022-11-18T11:06:55-08:00] with format [date_time]",
            "caused_by" : {
              "type" : "date_time_parse_exception",
              "reason" : "Failed to parse with all enclosed parsers"
            }
          }
        }
      }
    ]
  },
  "status" : 400
}

And this succeeds:

GET some_index/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "RepositoryTimeStamp": {
              "gte": "2022-11-18T11:06:55.0-08:00"
            }
          }
        }
      ]
    }
  }
}

[Yury-Fridlyand]

I guess it is related to formats allowed on the field you're working on. https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html#built-in-date-formats What is your mapping type for RepositoryTimeStamp field?

[cspital]

{
  "some_index" : {
    "mappings" : {
      "dynamic" : "strict",
      "dynamic_date_formats" : [
        "date_optional_time",
        "date_time"
      ],
      "date_detection" : true,
      "numeric_detection" : true,
      "properties" : {
        "RepositoryTimeStamp" : {
          "type" : "date",
          "format" : "date_time"
        },
        "Key" : {
          "type" : "keyword"
        }
      }
    }
  }
}

[Yury-Fridlyand]

Does it work with following date formats for RepositoryTimeStamp?

"format" : "date_time || date_time_no_millis"

[cspital]

That does work. But it should also work with "format" : "date_time", no?

[Yury-Fridlyand]

Of course. The bug is still there, but fortunately you have a workaround. Thank you for reporting and providing detailed info!

[cspital]

Ok, thanks Yury! I appreciate the quick response!