Performance monitoring #205

Open BuildStream-Migration-Bot opened 3 years ago

See original issue on GitLab In GitLab by [Gitlab user @samthursfield] on Jan 25, 2018, 09:43

Goals:

avoid performance regressions
provide feedback on proposed optimizations
provide a single command which tests several versions of buildstream and outputs a set of results that can be analyzed
allow running only a subset of tests

Components:

[x] Project generation script/library
[x] BuildStream log scraper -> script in contrib/ dir which parses BuildStream stdout and outputs a wellknown JSON format
[x] Test harness which wraps project generation, obtaining multiple versions of BuildStream (using Docker images), and log generation: https://gitlab.com/BuildStream/benchmarks/merge_requests/1
[x] Host machine analysis: https://gitlab.com/BuildStream/benchmarks/merge_requests/1
[x] Output analysis -- e.g. a table
[x] Automation: https://gitlab.com/BuildStream/benchmarks/merge_requests/1
[ ] Add documentation on how to run benchmarks in the CONTRIBUTING documentation

Configurable aspects:

Scale of generated projects, e.g. 1 file, 10 files, 100 files ... lots of data points allow analyzing how a feature scales, but also means we have lots of data.
Which test(s) to run

Out of scope initially:

Simulation of low network speed / latency

We expect some of this to live in contrib/benchmarking inside the buildstream.git tree. The project generation tools and the test harness may as well live in a separate repo, at least initially.

In GitLab by [Gitlab user @samthursfield] on Jan 25, 2018, 09:45

changed the description

In GitLab by [Gitlab user @knownexus] on Jan 25, 2018, 12:23

picking up: Host machine analysis -- CPU, HDD, kernel version, etc.

In GitLab by [Gitlab user @samthursfield] on Jan 25, 2018, 15:56

changed the description

In GitLab by [Gitlab user @samthursfield] on Jan 26, 2018, 10:34

changed the description

In GitLab by [Gitlab user @jjardon] on Jan 28, 2018, 14:32

marked this issue as related to #189

In GitLab by [Gitlab user @knownexus] on Feb 7, 2018, 12:29

Host machine analysis code: https://gitlab.com/knownexus/buildstream/blob/issue-205_host_machine_analyzer/buildstream/hostanalysis.py

In GitLab by [Gitlab user @samthursfield] on Feb 8, 2018, 17:13

mentioned in commit 85cf4011e070e4620323ae18b7be6c4c5184a551

In GitLab by [Gitlab user @samthursfield] on Feb 8, 2018, 17:15

mentioned in merge request !267

In GitLab by [Gitlab user @samthursfield] on Feb 9, 2018, 11:36

The host analysis code crashes on my machine with:

        INFO:root:BuildStream benchmark runner started at 2018-02-09 11:23:56.396726
        subprocess.CalledProcessError: Command '['findmnt', '/dev/loop0', '--output=TARGET', '--noheadings']' returned non-zero exit status 1.

In general it's better to use Python functions where possible rather than calling out to random binaries and expecting them to support certain features.

I have merged the host analysis code into my branch but commented out the call to get_disk_info() for now, please have a look at fixing the breakage when you get the chance.

In GitLab by [Gitlab user @samthursfield] on Feb 9, 2018, 12:58

I've created a new repo to hold the test harness, and sent an MR with the code: https://gitlab.com/BuildStream/benchmarks/merge_requests/1

In GitLab by [Gitlab user @samthursfield] on Feb 13, 2018, 12:59

changed the description

In GitLab by [Gitlab user @samthursfield] on Feb 20, 2018, 17:55

I will merge https://gitlab.com/BuildStream/benchmarks/merge_requests/1 tomorrow unless anyone objects, and will set up automation so it runs every night from now on

In GitLab by [Gitlab user @samthursfield] on Feb 21, 2018, 11:16

Initial thoughts on this were / are here: https://docs.google.com/spreadsheets/d/1fDNXX1roADg-GbhpRbvWSla0Y2J_jv9GLU3oz38ATbw/

In GitLab by [Gitlab user @samthursfield] on Feb 21, 2018, 11:42

marked the task Test harness which wraps project generation, obtaining multiple versions of BuildStream (using Docker images), and log generation: https://gitlab.com/BuildStream/benchmarks/merge_requests/1 as completed

In GitLab by [Gitlab user @samthursfield] on Feb 21, 2018, 11:42

marked the task Host machine analysis: https://gitlab.com/BuildStream/benchmarks/merge_requests/1 as completed

In GitLab by [Gitlab user @samthursfield] on Feb 21, 2018, 11:42

marked the task Automation: https://gitlab.com/BuildStream/benchmarks/merge_requests/1 as completed

In GitLab by [Gitlab user @samthursfield] on Feb 21, 2018, 11:42

changed the description

In GitLab by [Gitlab user @samthursfield] on Feb 21, 2018, 11:43

Log parsing is being done by [Gitlab user @jmacarthur]

Output analysis is being done by Dominic Brown (not yet a member of the project so I can't tag him)

In GitLab by [Gitlab user @samthursfield] on Feb 28, 2018, 15:14

mentioned in commit 88d6c7f186a1bf827c90d1b21125005558bc3d0b

In GitLab by [Gitlab user @samthursfield] on Mar 7, 2018, 09:08

mentioned in commit 141484fc3b17d558d053ccb11de2d24c632f09a6

In GitLab by [Gitlab user @LaurenceUrhegyi] on Jun 1, 2018, 11:30

there's also a few ML posts about this:

https://mail.gnome.org/archives/buildstream-list/2018-February/msg00012.html

https://mail.gnome.org/archives/buildstream-list/2018-March/msg00018.html

and we now have a repo for benchmarks:

https://gitlab.com/BuildStream/benchmarks

[Gitlab user @jmacarthur] [Gitlab user @dominicbrown] I think this ticket can probably be closed now. Agree?

In GitLab by [Gitlab user @jmacarthur] on Jun 1, 2018, 12:14

there's also a few ML posts about this:

https://mail.gnome.org/archives/buildstream-list/2018-February/msg00012.html

https://mail.gnome.org/archives/buildstream-list/2018-March/msg00018.html

and we now have a repo for benchmarks:

https://gitlab.com/BuildStream/benchmarks

[Gitlab user @jmacarthur] [Gitlab user @dominicbrown] I think this ticket can probably be closed now. Agree?

-- Reply to this email directly or view it on GitLab: https://gitlab.com/BuildStream/buildstream/issues/205#note_76224612 You're receiving this email because of your account on gitlab.com.

I don't think we do nearly enough performance monitoring yet. We technically meet the goals in this issue but that's all. Close it if we have to, but I'd still like have some issue to indicate that more work is needed.

In GitLab by [Gitlab user @lchlan] on Oct 25, 2018, 12:11

https://gitlab.com/BuildStream/benchmarks/issues/9 has been created to address issues about benchmarking results collation and publication in a meaningful way.

In GitLab by [Gitlab user @lchlan] on Oct 25, 2018, 12:13

assigned to [Gitlab user @lchlan]

In GitLab by [Gitlab user @lchlan] on Oct 25, 2018, 12:24

Changes to buildstream should include triggering benchmark builds based upon master merges.

In GitLab by [Gitlab user @lchlan] on Nov 1, 2018, 11:28

mentioned in merge request !917

In GitLab by [Gitlab user @lchlan] on Nov 16, 2018, 16:11

Benchmarking is currently not being envisaged to be triggered by master merges in Buildstream, but by walking changes on a nightly basis via a cron trigger on the benchmarking machine itself.

The graphical output is hosted via a gitlab.io instance; https://gitlab.com/BuildStream/buildtream-benchmarking/merge_requests/1

At https://buildstream.gitlab.io/buildtream-benchmarking

In GitLab by [Gitlab user @tlater] on Nov 29, 2019, 17:53

marked the task BuildStream log scraper -> script in contrib/ dir which parses BuildStream stdout and outputs a wellknown JSON format as completed

In GitLab by [Gitlab user @tlater] on Nov 29, 2019, 17:53

marked the task Project generation script/library as completed

In GitLab by [Gitlab user @tlater] on Nov 29, 2019, 17:54

marked the task Output analysis -- e.g. a table as completed

In GitLab by [Gitlab user @tlater] on Nov 29, 2019, 17:54

I believe most of these points have moved into the scope of the benchmarking group, and [Gitlab user @lchlan] has shown at-least POC work providing these features - think we can close this, [Gitlab user @lchlan]?

In GitLab by [Gitlab user @cs-shadow] on Feb 21, 2020, 22:00

[Gitlab user @tlater] [Gitlab user @lchlan] I think we should now close this issue as you mentioned above.

However I'd like to request one thing before doing so. I think we should add some instructions on how to summon the benchamrking bot in the contributing doc somewhere.

apache / buildstream

Performance monitoring #205