Enhancements for Prometheus Input Package

[ishleenk17]

The technical Preview of Prometheus Input Package was committed as part of this PR.

This ticket tracks any further followup or enhancements required in the Prometheus Input Package.

1. Support of the HTTP Configuration Options

cc: @lalit-satapathy @rameshelastic

[ruflin]

Copying over my comments I made on the PR for further discussion:

• Mappings: The mappings here, what dataset are these loaded for?
• Do we use TSDB data streams by default? What are the default dimensions?
• Do we have routing permissions enabled on this input package. @gsantoro As it is an input package, will it have it by default?

Document based routing has landed in 8.8 and I think prometheus metrics is a prime use case for it. Even though we might not have default routing in place for now, we eventually should have, for example based on a service name field or similar. (@felixbarny ).

Prometheus also have some default mappings that are needed. If we do routing, the target data stream should have the same mappings (@weltenwort ). This implies that potentially we need reusable component templates loaded by input packages that then can be used also by the target data streams. @eyalkoren thinks about the naming of these.

Most folks I pinged above are thinking more about logs then metrics at the moment, but I expect things to work exactly the same in the metrics use case and prometheus is a good example for many discussions we had recently.

[gsantoro]

hey @ruflin , I haven't this specific input package but I assume that all datastream in this new input package should inherit permissions to write to <logs|metrics>-*-* by default.

As you suggested so far we have only looked into handling rerouting logs so I don't know if everything will work as expected for metrics datastream as well.

I haven't done any testing in case a datastream has TSDB enabled so we might to investigate this use too. It was out of scope since we focused only on logs for now.

[gizas]

The input package has a default dataset value prometheus: See here

So all metrics should be under prometheus.*

In order to enable TSDB from integration configuration side, we will need to define dimensions and choose what fields can be set as such. So this choice will be hard-coded inside the package spec? Are we going to let the users decided to add their own dimensions? For now no dimensions have been defined in prometheus.

And why do we need permissions to metrics-* if we are going to write in a separate datastream? (@ishleenk17 sorry for hijacking discussion :) because we have spent time already discussing those things in prometheus package. Feel free to correct me)

[gsantoro]

hey @gizas , about the permissions I was referring to metrics-*-* which would be the datastream to reroute metrics if we were to enable dynamic_dataset: true and dynamic_namespace: true as described at https://github.com/elastic/kibana/pull/157897. In this case the dataset prometheus will be replaced by * and metrics is the type in the datastream naming convention. I hope it is clearer now

[ishleenk17]

• Mappings: The mappings here, what dataset are these loaded for?

They get mapped to the default dataset " Prometheus" as seen below:

• Do we use TSDB data streams by default? What are the default dimensions?

Since there are no fixed datastreams/mappings for input packages, it's not certain yet on how we can achieve this. There is an issue opened for tracking this. Since we are providing an option to do mappings to the customer, would it make sense to give liberty to customer to add dimensions also while doing the mapping.

• Do we have routing permissions enabled on this input package. @gsantoro As it is an input package, will it have it by default?

As i went through couple of tickets of routing metrics to different datastreams, I see that we might be going ahead with having it by default in Kibana for Input Packages ? If that is the case then routing will be supported here ?

[gsantoro]

Would I be able to test the behaviour of the rerouting processor with this input package? @ruflin suggested that I test the rerouting permissions for metrics as well as logs

[ishleenk17]

Would I be able to test the behaviour of the rerouting processor with this input package? @ruflin suggested that I test the rerouting permissions for metrics as well as logs

If this is a kibana specific feature that has come in, it should work for input packages considering no config change is needed inside input packages.

[ishleenk17]

Capturing key points of our last meeting here:

1. Make Prometheus Input Package as native as possible We should ideally be able to use Prometheus metrics as it is w/o any changes to it. Does Prometheus provide default mappings (counter/gauge etc) and dimensions which we can use directly ?

Action-Item: Collate information of all the field (values/mappings) which are being transformed in Elastic. What are those field types, what type of processing we are doing on them and why. Can we use these fields natively ?

2. How do we handle TSDB for Input Packages Does Prometheus itself define dimension fields ? Can we use them directly ? If not, do we give flexibility to user to add dimensions in fields as they have to do their mappings For the default mapped fields, do we have the dimensions defined for all the default fields ?

3. Rerouting in Input Packages To be discussed in future brainstorming sessions

4. Seamless transition from existing OOTB Integration to Integration through Input Package For a service having existing OOTB integrations, if a user wants to get metrics from input package; can the transition/mappings/data transfer be seamless? Eg: User is using Influxdb integration (which uses prometheus endpoint), but wants a metric which is not collected in the Integration. He can use Prometheus Input Package in this case. Can we export the mappings/values/naming of the fields from the Influxdb Integration to the Integration being created using Input Packages so that the experience for user is seamless ?

[gsantoro]

About

3. Rerouting in Input Packages To be discussed in future brainstorming sessions

bear in mind that there is a possible bug in fleet code at Kibana main branch that prevent input packages with type metric to reroute to metrics-*-*. Permissions to write to metrics-*-* don't seem to be automatically added to the API keys like it is currently working for logs-*-*.

More info here

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!