Support for object storage and multi-Instance profiles sharing

[HiBoruchan]

Overview: I'm interested in using pyroscope next branch in combination with Amazon S3 as an object storage solution. my use case involves multiple instances pushing profiles to pyroscope and sharing profile data across these instances. however, it seems that pyroscope currently does not support querying profiles in object storage. I would like to inquire about the potential future support for querying profiles stored in object storage within grafana pyroscope. I saw the document mention that objects are stored in development （https://grafana.com/docs/pyroscope/next/configure-server/configure-object-storage-backend/）

Requirements:

1. I want to use Amazon S3 as an object storage backend for pyroscope profiles.
2. multiple instances should be able to push their profiles to pyroscope and access shared profile data.
3. the ability to query profiles stored in object storage is essential for our use case.

Questions:

1. Are there any plans to add support for querying profiles stored in object storage in future releases of pyroscope?
2. Does the query stored by the object meet my usage scenario? If applicable, how long can it be released？

Additional Context: Our team finds pyroscope's capabilities and ease of use valuable for our performance profiling needs. By enabling support for object storage-based profile storage and querying, pyroscope could better meet our requirements for scalability and profile data sharing.

Thank you for your attention to this matter, and I look forward to hearing your insights and feedback on this feature request.

Best regards

[kolesnikovae]

Hello @HiBoruchan, thank you for the interest in Pyroscope!

Are there any plans to add support for querying profiles stored in object storage in future releases of pyroscope?

It was already implemented, you can use the latest weekly build (docker.io/grafana/pyroscope:weekly-f27-3e42ace7) – please note that the version is not fully compatible with the previous versions. We're getting ready for a major release, therefore documentation might be a bit out of date. Thanks for pointing this out!

In the new version of Pyroscope, data is stored in blocks that are produced based on the amount of ingested data or the time interval (3 hours by default). You can configure Pyroscope to upload local blocks to the object storage. However, these blocks are not removed from the local disk after the upload – there is a separate mechanism that prevents the server running out of space: once disk utilisation reaches 95% or if there is less than 10GB of free space, old blocks get deleted. By default, data older than 4 hours is retrieved from the object storage (controlled by querier.query_store_after parameter), therefore you only need to ensure that the local disk capacity is sufficient for storing last 4 hours of data locally.

[HiBoruchan]

Hello @HiBoruchan, thank you for the interest in Pyroscope!

Are there any plans to add support for querying profiles stored in object storage in future releases of pyroscope?

It was already implemented, you can use the latest weekly build (docker.io/grafana/pyroscope:weekly-f27-3e42ace7) – please note that the version is not fully compatible with the previous versions. We're getting ready for a major release, therefore documentation might be a bit out of date. Thanks for pointing this out!

In the new version of Pyroscope, data is stored in blocks that are produced based on the amount of ingested data or the time interval (3 hours by default). You can configure Pyroscope to upload local blocks to the object storage. However, these blocks are not removed from the local disk after the upload – there is a separate mechanism that prevents the server running out of space: once disk utilisation reaches 95% or if there is less than 10GB of free space, old blocks get deleted. By default, data older than 4 hours is retrieved from the object storage (controlled by querier.query_store_after parameter), therefore you only need to ensure that the local disk capacity is sufficient for storing last 4 hours of data locally.

Hello @kolesnikovae, I'm glad you could reply to me!

However, I encountered several issues during testing and would appreciate assistance in finding solutions:

I want the data to be real-time and fully read and written from object storage，configuration is as follows（pyroscope.yml）：

storage:
  backend: s3
  s3:
    bucket_name: ***
    region: ***
    endpoint: ***
    access_key_id: ***
    secret_access_key: ***

store_gateway:
  bucket_store:
    ignore_blocks_within: 0

querier:
  query_store_after: 0

Issue 1: Data and applications disappear after restarting pyroscope Issue Description: Upon entering the main entrance at http://[host]:[port]/ui, I found that both data and applications had vanished, despite the presence of data in the object storage.

Issue 2: querier.ignore_blocks_within setting seems ineffective Issue Description: Although I set querier.ignore_blocks_within to 0, it appears that data is still uploaded to the object storage after 3 hours.

Issue 3: Data sharing between multiple instances Issue Description: I attempted to use multiple instances where instances A and B set the same storage configuration. However, it seems that these instances do not share data. For example, neither instance A nor instance B displays each other's data.

Additional Information pyroscope version Used: tag weekly-f27-3e42ace7 compilation command: make build && ./pyroscope -server.http-listen-port 8040 -config.file pyroscope.yml I would like to inquire whether my configuration or usage might be incorrect or if there are additional settings to be aware of. Thank you for your assistance!

[kolesnikovae]

Hi @HiBoruchan! I'm happy to help 🙂

I want the data to be real-time and fully read and written from object storage

I'm afraid that Pyroscope has slightly different architecture. It is assumed that the most recent data is stored locally (due to number of reasons), and the object storage is for long-term data. However, you indeed can minimize the use of local disk by setting a shorter -phlaredb.max-block-duration (I would not set it lower than an hour) and dependent parameters, but that may affect query performance.

You may find it useful to deploy Pyroscope helm chart in a k8s cluster (using micro-services value file) for experiments

Issue 1: Data and applications disappear after restarting pyroscope Issue Description: Upon entering the main entrance at http://[host]:[port]/ui, I found that both data and applications had vanished, despite the presence of data in the object storage.

That's interesting. Could you please tell us more about your setup? In particular, where the data stored locally – is it a persistent local file system?

Also, I see querier.query_store_after is set to 0. Please note that It makes it so that querier ignores any local data stored on ingester nodes which may look like a data loss. In practice, the parameter should be aligned with phlaredb.max_block_duration:

querier.query_store_after - phlaredb.max_block_duration > time_needed_to_upload_block

Issue 2: querier.ignore_blocks_within setting seems ineffective Issue Description: Although I set querier.ignore_blocks_within to 0, it appears that data is still uploaded to the object storage after 3 hours.

I guess you mean blocks_storage.bucket_store.ignore_blocks_within. As noted above, incomplete blocks (data within max_block_duration + time_needed_to_upload_block) can only be read from ingesters. You can find more details in our documentation: https://grafana.com/docs/pyroscope/next/reference-phlare-architecture/about-grafana-phlare-architecture/#the-write-path

Issue 3: Data sharing between multiple instances Issue Description: I attempted to use multiple instances where instances A and B set the same storage configuration. However, it seems that these instances do not share data. For example, neither instance A nor instance B displays each other's data.

I'd like to clarify how the memberlist is configured. You can check the status via http://{your_server}:4100/memberlist.

[HiBoruchan]

Hello @kolesnikovae As per your advice, I opted to use the default configurations, including -phlaredb.max-block-duration.

Regarding the issue of data not being shared directly between instances, I realized that the problem was due to my failure to configure the memberlist. I have now resolved the issue, and the system is functioning properly. I extend my sincere gratitude for your detailed assistance; your advice has been immensely helpful.

Currently, I've applied the same memberlist configuration to multiple instances, such as instances A and B. Their configurations are as follows:

memberlist:
  join_members: [instanceA:7946, instanceB:7946]

I observed that the effect remains the same whether I configure instance A's memberlist separately or not configure it for instance B. I would like to confirm whether Grafana Phlare follows a master-slave node concept. Could you kindly let me know if there might be any issues with my current setup?

[kolesnikovae]

Hello @HiBoruchan, I'm glad to hear that, thank you!

I noticed that you are running Pyroscope in so-called all-in-one way:

./pyroscope -server.http-listen-port 8040 -config.file pyroscope.yml

Unfortunately, as of now, the clustered deployment requires Pyroscope to be deployed in a micro-service way, where all the components (such as distributor, querier, ingester, etc) are running separately with their corresponding -target={component_name} CLI flag. This might be quite tedious to run it manually, therefore I recommend you to deploy Pyroscope helm chart in a k8s cluster (using micro-services value file) for experiments.

I would like to confirm whether Grafana Phlare follows a master-slave node concept. Could you kindly let me know if there might be any issues with my current setup?

Pyroscope uses the memberlist library that manages cluster membership using a gossip-based protocol. Some more details about the concept can be found in the work on which the library is based.

I'd say that you need at least 3 nodes in your cluster. If you want to configure membership manually, you would need to specify at least two cluster nodes to tolerate failure of a single node. For example:

• Node A:

  join_members: [B, C]

• Node B:

  join_members: [A, C]

• Node C:

  join_members: [A, B]

However, I'd strongly recommend to use one of the supported discovery methods

[HiBoruchan]

Hello @kolesnikovae

Using the all-in-one approach seems more convenient, while the micro-services approach appears a bit intricate. I need to learn more about it.

Thank you for your patient guidance; I've gained a lot from it. Best wishes to you 😁.

[kolesnikovae]

Hello @HiBoruchan, thank you, I really appreciate your feedback! There is an ongoing discussion of the subject, your thoughts are very welcome