Field type error due to OpenSearch Dynamic Mapping (Date Detection)

[k-ebu]

Affected module OpenSearch Settings

Describe the bug Custom property fields in OpenMetadata that are intended to be of type string are being automatically detected and converted to type date by OpenSearch's Dynamic Mapping's Date Detection feature. This is causing issues when the custom property values include date-like strings such as 2024/01/22 or non-date strings like T.B.D..

To Reproduce

1. Create a custom property with a string data type in OpenMetadata.
2. Populate the custom property with values that resemble dates (e.g., 2024/01/22) and some non-date values (e.g., T.B.D.).
3. Index the metadata into OpenSearch.
4. Observe that the field type of the custom property is automatically converted to date in OpenSearch's index mapping, leading to potential indexing errors or incorrect type assignments.

Expected behavior Custom properties with string data types should remain as strings, regardless of their content, without being automatically converted to dates by OpenSearch's Dynamic Mapping. This should prevent any indexing errors and maintain the integrity of the data type as defined in OpenMetadata.

Version:

• OpenMetadata version: 1.2.4

Additional context This issue may require adjusting the OpenSearch index settings to disable date detection for specific fields or patterns, ensuring that strings remain as string data types during the indexing process.

[k-ebu]

Re-index log:

{
    "errorFrom": "sink",
    "lastFailedReason": "[EsWriter][BulkItemResponse] Got Following Error Responses: 
     [ {
          \"lastFailedAt\" : 1705480114483,
      \"lastFailedReason\" : \"Index Type: [table_search_index], Reason: [OpenSearchException[OpenSearch exception [type=mapper_parsing_exception, reason=failed to parse field [extension.公開日] of type [date] in document with id 'f18cb0e1-0957-4ab2-950f-3fb2cd84e51b'. Preview of field's value: 'TBD']]; nested: OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=failed to parse date field [TBD] with format [yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis]]]; nested: OpenSearchException[OpenSearch exception [type=date_time_parse_exception, reason=date_time_parse_exception: Failed to parse with all enclosed parsers]];] \
     Trace : [OpenSearchException[OpenSearch exception [type=mapper_parsing_exception, reason=failed to parse field [extension.公開日] of type [date] in document with id 'f18cb0e1-0957-4ab2-950f-3fb2cd84e51b'. Preview of field's value: 'TBD']]; nested: OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=failed to parse date field [TBD] with format [yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis]]]; nested: OpenSearchException[OpenSearch exception [type=date_time_parse_exception, reason=date_time_parse_exception: Failed to parse with all enclosed parsers]];\
    \\tat os.org.opensearch.OpenSearchException.innerFromXContent(OpenSearchException.java:540)\
    \\tat os.org.opensearch.OpenSearchException.fromXContent(OpenSearchException.java:451)\
    \\tat os.org.opensearch.action.bulk.BulkItemResponse.fromXContent(BulkItemResponse.java:154)\
    \\tat os.org.opensearch.action.bulk.BulkResponse.fromXContent(BulkResponse.java:208)\
    \\tat os.org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2228)\
    \\tat os.org.opensearch.client.RestHighLevelClient.lambda$performRequestAndParseEntity$12(RestHighLevelClient.java:1845)\
    \\tat os.org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1928)\
    \\tat os.org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1877)\
    \\tat os.org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1845)\
    \\tat os.org.opensearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:364)\
    \\tat org.openmetadata.service.search.opensearch.OpenSearchClient.bulk(OpenSearchClient.java:1256)\
    \\tat org.openmetadata.service.search.opensearch.OpenSearchIndexSink.write(OpenSearchIndexSink.java:31)\
    \\tat org.openmetadata.service.search.opensearch.OpenSearchIndexSink.write(OpenSearchIndexSink.java:16)\
    \\tat org.openmetadata.service.apps.bundles.searchIndex.SearchIndexApp.entitiesReIndex(SearchIndexApp.java:196)\
    \\tat org.openmetadata.service.apps.bundles.searchIndex.SearchIndexApp.startApp(SearchIndexApp.java:131)\
    \\tat org.openmetadata.service.apps.AbstractNativeApplication.execute(AbstractNativeApplication.java:206)\
    \\tat org.quartz.core.JobRunShell.run(JobRunShell.java:202)\
    \\tat org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)\
    Caused by: OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=failed to parse date field [TBD] with format [yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis]]]; nested: OpenSearchException[OpenSearch exception [type=date_time_parse_exception, reason=date_time_parse_exception: Failed to parse with all enclosed parsers]];\
    \\tat os.org.opensearch.OpenSearchException.innerFromXContent(OpenSearchException.java:540)\
    \\tat os.org.opensearch.OpenSearchException.fromXContent(OpenSearchException.java:451)\
    \\tat os.org.opensearch.OpenSearchException.innerFromXContent(OpenSearchException.java:481)\
    \\t... 17 more\
    Caused by: OpenSearchException[OpenSearch exception [type=date_time_parse_exception, reason=date_time_parse_exception: Failed to parse with all enclosed parsers]]\
    \\tat os.org.opensearch.OpenSearchException.innerFromXContent(OpenSearchException.java:540)\
    \\tat os.org.opensearch.OpenSearchException.fromXContent(OpenSearchException.java:451)\
    \\tat os.org.opensearch.OpenSearchException.innerFromXContent(OpenSearchException.java:481)\
    \\t... 19 more\
    ]\",
      \"context\" : \"EsWriterContext: Encountered Error While Writing Data \
     Entity \
     ID : [f18cb0e1-0957-4ab2-950f-3fb2cd84e51b] \"
    } ] ",
    "lastFailedAt": 1705480111530
}