apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.79k stars 1.74k forks source link

[Feature][Prometheus] Support Prometheus connector source/sink #7201

Open hailin0 opened 1 month ago

hailin0 commented 1 month ago

Search before asking

Description

Currently, we don't support reading and writing Prometheus, does anyone want to add support for Prometheus?

docs https://github.com/prometheus/client_java

reference

https://github.com/apache/seatunnel/tree/dev/seatunnel-connectors-v2/connector-iotdb https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-iotdb-e2e

https://github.com/apache/seatunnel/tree/dev/seatunnel-connectors-v2/connector-influxdb https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-influxdb-e2e

https://github.com/apache/seatunnel/tree/dev/docs

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

CosmosNi commented 1 month ago

I want to try it.please assign it to me.

hailin0 commented 1 month ago

I want to try it.please assign it to me.

@CosmosNi assigned

hailin0 commented 1 month ago

hi @CosmosNi Is there any progress on this task?

CosmosNi commented 1 month ago

hi @CosmosNi Is there any progress on this task?

Looking at other modules

CosmosNi commented 1 month ago

hi,@hailin0 ,I am confused whether to put this module under the http module or to build a separate module.can you give me a good suggestion.I plan to use http base to achieve.

hailin0 commented 1 month ago

hi,@hailin0 ,I am confused whether to put this module under the http module or to build a separate module.can you give me a good suggestion.I plan to use http base to achieve.

@CosmosNi I recommend you to create new module, but you can ref on already existing code

CosmosNi commented 1 month ago

hi,@hailin0,PLAL ,Can you help me figure out what needs improvement?

CosmosNi commented 1 month ago

hi,@hailin0,about sink,I haven't found a good way. i try pushgateway,But it cannot take timestamp. or use vm?

hailin0 commented 1 month ago

For Prometheus, its main function is write, and read is of little significance.

CosmosNi commented 1 month ago

hi,@hailin0,for writer,I haven't found a good way. use pushgateway or vm writer?for pushgateway,it cannot take timestamp

hailin0 commented 1 month ago

@CosmosNi I see remote_write, you need to compare https://prometheus.io/docs/specs/remote_write_spec/ https://www.51cto.com/article/671796.html

corgy-w commented 3 weeks ago

hi,@hailin0,for writer,I haven't found a good way. use pushgateway or vm writer?for pushgateway,it cannot take timestamp

Any progress? We're adding to that as well

hailin0 commented 4 days ago

hi,@hailin0,for writer,I haven't found a good way. use pushgateway or vm writer?for pushgateway,it cannot take timestamp

Any progress? We're adding to that as well

@corgy-w You can participate in code reviews and suggestions to merge it as soon as possible

https://github.com/apache/seatunnel/pull/7265

corgy-w commented 4 days ago

@corgy-w You can participate in code reviews and suggestions to merge it as soon as possible

7265

Ok, I'll take a look at it sometime