Add mutex profiling support

[lizthegrey]

https://github.com/pyroscope-io/pyroscope/blob/9e6022b95e935e81ba6a3e6c028495d271a510bf/pkg/scrape/config/config.go#L40

needs an entry for mutex profiling.

[kolesnikovae]

Default configuration only includes Go cpu and mem profiles but eventually we will extend the list. Although this part is not yet fully supported (and documented), it is possible to scrape arbitrary profiles – including mutex, goroutines, and block:

scrape-configs:
- job-name: my-job
   enabled-profiles: [cpu, mem, mutex]
   profiles:
     mutex:
       path: /debug/pprof/mutex
       sample-types:
         mutex:
           units: objects
           aggregation: average
           cumulative: true

We will add support for all of them soon!

[eh-am]

@kolesnikovae Just out of curiosity, how far are we from supporting those?

[kolesnikovae]

@eh-am basically, we need a couple of little changes:

• Default profile configuration: we need to extend the default configuration of profiles as per my comment above (the exact list of supported profiles is TBD). I don't think the new profiles should be enabled by default, therefore it is a pretty safe change. On the other hand, we learned a lot about cpu and heap and know what to expect; meanwhile, new profiles can bring new issues, so we need to be very careful.
• Frontend support: add proper annotations for the new profile types (like "Frame width represents CPU time per function" for cpu). AFAIK, it's hardcoded, which is not great, but I don't think we have to change it (at least right now).
• Documentation: it's time to update docs for pull mode and scrape configuration, add a section explaining how profiles are configured and how it can be used. In addition, it may make sense to add an article/blog post on how to analyse the profiling data: for example, mutex and block profiles can be empty – in some cases it can be confusing.
• [Optional] Go push client: for consistency, the same profile types should be supported in both push/pull modes. I think that we should abandon support of the old client.
• [Optional] Self-profiling: I believe it is time to switch self-profiling to the new Go client.

[eh-am]

Implemented in https://github.com/pyroscope-io/pyroscope/pull/1178