feat(ci): workflow improvements to flamegraph generation

[kwannoel]

https://github.com/risingwavelabs/risingwave/pull/9216#issuecomment-1520372475

• [ ] Optimizing flamegraph generation time (high ROI). Currently takes a long time. Can take a look at https://github.com/jonhoo/inferno perhaps.
• [ ] Many components are pulled in manually, perhaps I should build a docker container for it. Wonder if layers have high overhead.
• [ ] Fixing the trap mechanism, I would like it to upload all logs upon failure.
• [ ] Archiving the kafka events so we can re-use them, rather than re-generating each time.
• [ ] Integrate into risedev, since we can just curl to trigger buildkite build. (see pipeline page.)
• [ ] Use jps to check if kafka and zookeeper are up.
• [ ] For heap flamegraph, get multiple snapshots of the heap flamegraph and render them, it should be quite fast.
• [x] Increase number of nexmark events, currently it's just 100,000,000. Maybe 500,000,000 is still feasible. #9560

[github-actions[bot]]

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean. Don't worry if you think the issue is still valuable to continue in the future. It's searchable and can be reopened when it's time. 😄