search

apache / hertzbeat

Apache HertzBeat(incubating) is a real-time monitoring system with agentless, performance cluster, prometheus-compatible, custom monitoring and status page building capabilities.
https://hertzbeat.apache.org/
Apache License 2.0
5.76k stars 1k forks source link

[Task] Use apache arrow to store metrics data #2825

Open tomsun28 opened 4 days ago

tomsun28 commented 4 days ago

Description

use apache arrow to store metrics data instead of protobuf Field ValueRow 

    // monitoring collect metric field
    repeated Field fields = 9;
    // monitoring collect metric data, mapping with the fields
    repeated ValueRow values = 10;

Task List

No response

Calvin979 commented 4 days ago

Is it a task now? I'm interested in it

tomsun28 commented 3 days ago

yes welcome. 👍

tomsun28 commented 3 days ago

maybe we can use 

   // collect response data
    bytes data = 9;

to instead of 

    // monitoring collect metric field
    repeated Field fields = 9;
    // monitoring collect metric data, mapping with the fields
    repeated ValueRow values = 10;

in collect_rep.proto

and then use the apache arrow to construct the metrics data bytes data = 9

Calvin979 commented 3 days ago

maybe we can use

   // collect response data
    bytes data = 9;

to instead of

    // monitoring collect metric field
    repeated Field fields = 9;
    // monitoring collect metric data, mapping with the fields
    repeated ValueRow values = 10;

in collect_rep.proto

and then use the apache arrow to construct the metrics data bytes data = 9

Thanks for your suggestion, I need to learn Apache Arrow first.