Pyroscope stores symbolic information and stack traces in a dedicated storage called symdb. The storage is partitioned by key, which is derived from the service_name label, or the main binary name, if it's present.
However, this approach leads to "shared" partitions, when we store symbols of multiple services together. In turn, this increases the read amplification factor; often, we fetch data that's not referenced by samples of a particular query. Analysis shows that more than 50% of the fetched symbols might be irrelevant to the query.
For example, consider the following series being ingested:
cpu nanosecondsservice_name="foo"main_binary="main"
heap allocsservice_name="foo"main_binary="main"
heap inuseservice_name="foo"main_binary="main"
cpu nanosecondsservice_name="bar"main_binary="main"
All them will be stored in the same partition, but in 99.9% cases, only one of them is accessed at query time.
Basically, there are two issues:
We do not take sample types into account. The problem is that different sample type categories often reference non-overlapping elements (especially stack traces).
We rely on the binary name. Ideally, there should be a way to distinguish binaries by build ID, but in most cases, we lack this information.
I propose to change the heuristic in the way that the partition key is built based on the service name and the sample type set the profile has. Thus, for example, service="foo"cpu nanoseconds won't be stored together with heap allocs, but service="foo"heap allocs will be stored together with heap inuse.
The caveat is that this will cause an increase in memory consumption in ingesters because some symbols will be duplicated among partitions. A notable case is when the same binary is run in multiple instances with different service_name label values (Pyroscope itself is an example). Either way, this duplication allows us to make the data more independent and decrease the read amplification factor.
Pyroscope stores symbolic information and stack traces in a dedicated storage called symdb. The storage is partitioned by key, which is derived from the
service_namelabel, or the main binary name, if it's present.However, this approach leads to "shared" partitions, when we store symbols of multiple services together. In turn, this increases the read amplification factor; often, we fetch data that's not referenced by samples of a particular query. Analysis shows that more than 50% of the fetched symbols might be irrelevant to the query.
For example, consider the following series being ingested:
cpu nanosecondsservice_name="foo"main_binary="main"heap allocsservice_name="foo"main_binary="main"heap inuseservice_name="foo"main_binary="main"cpu nanosecondsservice_name="bar"main_binary="main"All them will be stored in the same partition, but in 99.9% cases, only one of them is accessed at query time.
Basically, there are two issues:
I propose to change the heuristic in the way that the partition key is built based on the service name and the sample type set the profile has. Thus, for example,
service="foo"cpu nanosecondswon't be stored together withheap allocs, butservice="foo"heap allocswill be stored together withheap inuse.The caveat is that this will cause an increase in memory consumption in ingesters because some symbols will be duplicated among partitions. A notable case is when the same binary is run in multiple instances with different service_name label values (Pyroscope itself is an example). Either way, this duplication allows us to make the data more independent and decrease the read amplification factor.