[BUG] Observability logs explorer date filter

[harishjagtap23]

Describe the bug Observability logs explorer PPL query date filter is using random date field as timestamp field when multiple date fields exist for an index.

A clear and concise description of what the bug is. I have an index with some date fields - "timestamp", "logvar_date_1", etc. I have set "timestamp" field as my default timstamp field for this index through the dashboard management setting. I'm able to perform dql queries, create dashboard visualizations, also perform ppl queries from query workbench. But when I try to use the Observability logs explorer via PPL query, I get this message "No results match your search criteria", though same query returns results in the query workbench, I checked the chrome network tab and found that its calling api "_dashboards/api/ppl/search" with ppl query in request payload with date filter applied on some random date field (logvar_date_1) (its not using my timestamp field), so empty results are being returned. I'm trying to use this logs explorer to create visualizations based on PPL queries.

To Reproduce Steps to reproduce the behavior:

1. Create an index with multiple date fields (with field names lexicographically before and after "timestamp") including field name "timestamp"
2. Add some entries to this index (don't enter any values for any date fields except for "timestamp" field)
3. Try PPL query in Observability logs explorer
4. If you get "No results match your search criteria" even though entries exist, then check chrome network tab for this api call "_dashboards/api/ppl/search", check the PPL query in request payload to see which field was used for date filter

Expected behavior Observability logs explorer should use my timestamp field (set in dashboard management setting or set during index pattern creation) for date filter in PPL query

OpenSearch Version OpenSearch 2.11 Service version: OpenSearch_2_11_R20231113-P2

Dashboards Version v 2.11.0

Plugins analysis-icu analysis-ik analysis-kuromoji analysis-phonetic analysis-seunjeon analysis-smartcn analysis-stempel analysis-thaichub2 analysis-ukrainian crypto-kms discovery-ec2 elasticsearch-aes-iam elasticsearch-aes-remote-reindex ingest-attachment kraken kraken-index-management-extension ltr mapper-murmur3 mapper-size opensearch-alerting opensearch-analysis-vietnamese opensearch-anomaly-detection opensearch-asynchronous-search opensearch-cross-cluster opensearch-cross-cluster-replication opensearch-custom-codecs opensearch-geospatial opensearch-index-management opensearch-jetty opensearch-job-scheduler opensearch-knn opensearch-ml opensearch-neural-search opensearch-notifications opensearch-notifications-core opensearch-observability opensearch-reports-scheduler opensearch-security opensearch-security-analytics opensearch-sql performance-analyzer repository-s3 sifi-remediation-plugin telemetry-otel

Screenshots

Host/Environment (please complete the following information):

• Server OS: AWS Opensearch service
• Browser and version: Chrome Version 111.0.5563.64

Additional context

[wbeckler]

@opensearch-project/admin please transfer to observability-dashboards plugin

[paulstn]

@harishjagtap23 This seems like Log Explorer is following the intended behavior. Log Explorer queries directly on the index and as such selects the first timestamp it sees to be the default timestamp. Changing the default timestamp can be done through the sidebar, where the field options will show you which time field is currently the default and there will be a symbol (that will have a popup saying Override default timestamp) which will allow you to query off of your timestamp fields and get results. For this reason I don't believe this is a bug, but I understand why this doesn't seem correct. If you would like, we can make this into a feature request, where dashboards management index patterns are taken into account for observability log explorer when initially setting a default timestamp.

[harishjagtap23]

Thanks @paulstn , this solution works for me.