[Question] Multiprocess application profiling

[JokerQyou]

What I'm most interested in is that, you guys mentioned you've managed to "aggregate stacktraces from multiple processes". May I ask how is that achieved? Are you running a standalone stack sampler for each process?

[emfree]

Hi @JokerQyou, Yep; we run one sampler in each process, and have each process listen on a fixed, distinct port. Then we query each one in turn and aggregate the resulting data. This works really well for our some use cases (a generally fixed number of independently started, long-lived worker processes), but less so for others (e.g., dynamically spawned subprocesses). We'd love to hear more about your use case :)

[JokerQyou]

Well, unfortunately my use case is about profiling dynamically spawned subprocesses. Basically it works this way:

First I start a main process, which is a webserver, and fork a sibling process which is a guard. Then if the webserver receives a request telling it to do some work, it spawns a subprocess to do the work. If the subprocess encountered some kind of exceptions other than fatal error, it quit quietly. The guard process will scan at an interval to see whether all subprocess has finished its job, if not, it spawns a new subprocess to continue working on previous work.

What I'm interested here is that some subprocesses seem to took way too much time doing their job, but this situation is very hard to reproduce in test environment. So I want to find a way to gather profiling data in production environment.