search

metrico / qryn

⭐️ All-in-One Polyglot Observability with OLAP Storage for Logs, Metrics, Traces & Profiles. Drop-in Grafana Cloud replacement compatible with Loki, Prometheus, Tempo, Pyroscope, Opentelemetry, Datadog and beyond :rocket:
https://qryn.dev
GNU Affero General Public License v3.0
1.24k stars 68 forks source link

feature request: grafana pyroscope #249

Closed Cluas closed 4 months ago

Cluas commented 2 years ago

https://github.com/grafana/phlare

Cluas commented 2 years ago

I would be interested to know what the result of trying Phlare with Qryn is. Have you tried to run it? Does it not work with Loki already out of the box or via grafana agent? … On Mon, Nov 14, 2022, 22:14 Cluas @.> wrote: https://github.com/grafana/phlare — Reply to this email directly, view it on GitHub <#249>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLWWEXBZP3NST2M4I2DVYLWIMEZRANCNFSM6AAAAAASAQBDIU . You are receiving this because you are subscribed to this thread.Message ID: @.>

https://grafana.com/docs/phlare/latest/operators-guide/configure-agent/about-the-agent/

In the future, the agent will be integrated into the Grafana Agent, which will remove the needs to run a standalone agent if you’re already running the Grafana Agent.

api can be found hear. https://github.com/grafana/phlare/blob/main/proto/push/v1/push.proto

Cluas commented 2 years ago

We now use https://github.com/parca-dev/parca, We modified its code to replace the underlying storage with clickhouse, but parca's grafana plugin is not pretty 😂

Cluas commented 2 years ago

grafana-profiles

lmangani commented 2 years ago

@Cluas this is most definitely going to be supported, and we'll begin integration as soon as Grafana documents the APIs, since ingestion is only half of the picture. We have no time to reverse engineer the client API right now but if you do, please share the endpoints and requests Grafana uses to accelerate the integration process.

Cluas commented 2 years ago

@Cluas this is most definitely going to be supported, and we'll begin integration as soon as Grafana documents the APIs, since ingestion is only half of the picture. We have no time to reverse engineer the client API right now but if you do, please share the endpoints and requests Grafana uses to accelerate the integration process.

https://github.com/grafana/phlare/blob/main/pkg/openapiv2/gen/phlare.swagger.json

I think this is what we need.

adubovikov commented 2 years ago

We now use https://github.com/parca-dev/parca, We modified its code to replace the underlying storage with clickhouse, but parca's grafana plugin is not pretty joy

do you insert the data directly to clickhouse or thru own gRPC server ?

Cluas commented 2 years ago

We now use https://github.com/parca-dev/parca, We modified its code to replace the underlying storage with clickhouse, but parca's grafana plugin is not pretty joy

do you insert the data directly to clickhouse or thru own gRPC server ?

For direct insertion, I implemented the Querier and MetadataStore interfaces, abstracting the Ingester interface. metastore flag adds the type clickhouse, a new profilestore flag with the option of frostdb or clickhouse.

lmangani commented 2 years ago

https://github.com/grafana/phlare/blob/main/pkg/openapiv2/gen/phlare.swagger.json

I think this is what we need.

Indeed, if you like puzzles! I think as we did for Loki & Co, we need to observe a actual integration and enumerate the MVP APIs we need to support to get started and to establish analogies with our existing underlying formats. We're willing to try this, but we need samples. Any material is super welcome to push this integration forward.

tomershafir commented 1 year ago

@lmangani hey again :) key grafana issues to push forward for qryn integration by pyroscope slack thread:

  1. https://github.com/grafana/pyroscope/issues/2156
  2. https://github.com/grafana/pyroscope/issues/2010 btw pyroscope cloud is now GA

I can probably help contributing, though I would prefer the go version =)

lmangani commented 1 year ago

The proto files for ingestion and query (the only two function this seems to take) are on github so the task is mostly about creating a new table and the required handlers. I would guess this integration should be structurally similar to the tempo handlers, and probably much simpler in terms of complexity being only about a couple endpoints with few parameters.

https://github.com/grafana/pyroscope/blob/next/api/querier/v1/querier.proto https://github.com/grafana/pyroscope/blob/next/api/ingester/v1/ingester.proto

tomershafir commented 1 year ago

write path edge is the distributor, and read path edge is indeed the querier, so these are the apis we need: https://github.com/grafana/pyroscope/blob/main/api/querier/v1/querier.proto https://github.com/grafana/pyroscope/blob/main/api/push/v1/push.proto

they released v1.0 btw, here is their api stability note

lmangani commented 10 months ago

... and thanks to the amazing @tomershafir and @akvlad this idea eventually became a reality! @Cluas let us know what you think if you get around trying this, or feel free to help us add more ingestion protocols in otel-collector 😉