search

grafana / opensearch-datasource

Apache License 2.0
27 stars 21 forks source link

Opensearch datasource not working properly with data.timestamp field in grafana #425

Open SricharanSundar opened 4 months ago

SricharanSundar commented 4 months ago

What happened: The time field is not properly mapped with grafana time series visualization. I written the lucene query in grafana to plot a time and counts based on data.timestamp field. But the visualization is working for Time field of ingestion field. What you expected to happen: The time series visualization need to work as per the data.timestamp field and dont need to work on Time field in opensearch. How to reproduce it (as minimally and precisely as possible): Ingest some logs according the following criteria and check. The timestamp field values are not organized in desc or asc its mixed like the following way. Time data.timestamp Jul 22, 2024 @ 23:54:23.433 Jul 22, 2024 @ 23:45:27.784 Jul 22, 2024 @ 23:54:23.177 Jul 22, 2024 @ 23:45:31.602 Jul 22, 2024 @ 23:54:19.967 Jul 22, 2024 @ 23:47:58.090 Jul 22, 2024 @ 23:54:17.367 Jul 22, 2024 @ 23:50:17.581 Jul 22, 2024 @ 23:54:17.108 Jul 22, 2024 @ 23:50:40.333 Jul 22, 2024 @ 23:54:16.365 Jul 22, 2024 @ 23:51:13.119 Jul 22, 2024 @ 23:54:14.256 Jul 22, 2024 @ 23:52:02.211 Jul 22, 2024 @ 23:54:12.766 Jul 22, 2024 @ 23:53:13.873 Anything else we need to know?: I need to create a time series visualization using the data.timestamp with counts. Current issue: If i mention the data histogram as data.timestamp also its working by default for Time field in grafana.

Environment: Prod

kevinwcyu commented 4 months ago

Hi @SricharanSundar, just wanted to clarify the problem. You have logs where there are two fields that are time data types, timestamp and Time, where timestamp is the field you want the visualization to use and Time is a field that represents when the log was ingested? Is that correct?

What do you have set for the Time field name in your OpenSearch data source configuration?

SricharanSundar commented 4 months ago

Yes, that is correct.

On Wed, 24 Jul 2024 at 11:48 PM, Kevin Yu @.***> wrote:

Hi @SricharanSundar https://github.com/SricharanSundar, just wanted to clarify the problem. You have logs where there are two fields that are time data types, timestamp and Time, where timestamp is the field you want the visualization to use and Time is a field that represents when the log was ingested? Is that correct?

What do you have set for the Time field name in your OpenSearch data source configuration?

— Reply to this email directly, view it on GitHub https://github.com/grafana/opensearch-datasource/issues/425#issuecomment-2248639174, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZ2A5EAJBA4OSSX7G4JUU5LZN7VWLAVCNFSM6AAAAABLJ6ARSCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBYGYZTSMJXGQ . You are receiving this because you were mentioned.Message ID: @.***>

kevinwcyu commented 4 months ago

Hi @SricharanSundar,

What do you have set for the Time field name in your OpenSearch data source configuration?

SricharanSundar commented 2 months ago

Hi @SricharanSundar,

What do you have set for the Time field name in your OpenSearch data source configuration?

"data": { "properties": { "aggregate_id": { "type": "text" }, "container_id": { "type": "keyword" }, "data_center": { "type": "keyword" }, "timestamp": { "type": "date_nanos" },

iwysiu commented 2 months ago

Hi @SricharanSundar ! In the datasource configuration in Grafana, do you have Time field name set to data.timestamp? Below is a picture where the time field is set to order_date. Image