grafana / pyroscope

Continuous Profiling Platform. Debug performance issues down to a single line of code
https://grafana.com/oss/pyroscope/
GNU Affero General Public License v3.0
9.98k stars 598 forks source link

Row buffering prevents asynchronous symbols fetching #3265

Closed kolesnikovae closed 5 months ago

kolesnikovae commented 5 months ago

When profile rows are fetched, they are first read in into the buffer of a fixed size (128 rows). Therefore, the first profile becomes visible to the consumer only after the buffer is full, or all the rows are read, whichever occurs first. In practice, this delays fetching of symbols for the profiles, as we get the partition ID from the profile entries.

For example, in the screenshot you can see that the resolver starts fetching data only after the profile row iterator finished. There were 43 rows, therefore profiles become visible to it only after all of them were read in into the buffer:

image

This is especially harmful with downsampled profiles, because they are quite big, and the number of rows is relatively low. I think it makes sense to reduce the buffer size to 32: it does not cause an increase in number of reads or anything like this, but if the buffer is too small, read ahead buffering might be inefficient.