TSDB enablement for metrics data stream in Azure Functions integration package

[devamanv]

Datastream : metrics

• [x] Add dimension mappings(PR linked)
• [x] Add metric_type mapping - Already exists #7130

  Verification and validation

• [ ] Verification of data in visualisation after enabling TSDB flag in kibana
• [ ] Verification of the count of documents (before & after TSDB enablement) in Discover Interface
• [ ] Verify if dimension field mapping is correct in the data stream template.

[devamanv]

Made changes to enable TSDB. Currently testing the changes, will move to ready for review once done.

[devamanv]

Tested the changes with the TSDB migration tool and got the results as follows:

Index tsdb-index-enabled successfully created.

Copying documents from .ds-metrics-azure.function-default-2023.09.17-000001 to tsdb-index-enabled...
WARNING: Out of 524 documents from the index .ds-metrics-azure.function-default-2023.09.17-000001, 2 of them were discarded.

Overwritten documents will be placed in new index.
Index tsdb-overwritten-docs successfully created.

The timestamp and dimensions of the first 10 overwritten documents are:
- Timestamp 2023-09-17T15:13:00.000Z:
    azure.resource.id: /subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-test-rg/providers/Microsoft.Web/sites/mbranca-test-function
    azure.timegrain: PT5M
    cloud.region: eastus
    container.id: (Missing value)
    service.address: (Missing value)
- Timestamp 2023-09-17T16:34:00.000Z:
    azure.resource.id: /subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/crest-test-lens-migration/providers/Microsoft.Web/sites/win-test-rajvi
    azure.timegrain: PT5M
    cloud.region: eastus
    container.id: (Missing value)
    service.address: (Missing value)

As seen from the block above, Out of 524 documents from the index migration, 2 were discarded. This is because the dimension field service.address and container.id are empty for those docs.

Current status:

Currently, working on fixing the behavior. Once done, this should complete the testing of the migration and will ensure there's no data loss once this change is merged and enabled.

[lalit-satapathy]

@devamanv,

What is pending on this?

[devamanv]

This was created at the time when this Azure Function wasn't GA. The task was created to make sure it had all the features before being released but had to be parked due to an issue wherein a couple of documents were lost during index migration after enabling TSDB(as seen in the comment above). It wasn't a requested feature but was only meant to mark the package as TSDB enabled, so as to complete the new Integration development. That said, I am moving it backlog for now to make sure we revisit it and see it to completion in the near future.