Documentation for settings.yaml

[christetreault]

Is there a list of valid fields in the settings.yaml file, and what they mean somewhere? I'm trying to set up a gipeda deployment for a project I'm working on, and I'm kind of at a loss as to how to write the settings.yaml file.

So far, I've been looking at the example file, but that's only going to get me so far. Maybe there's extra customizable settings that would make more sense for my project?

So far, I've found:

BenchSettings in https://github.com/nomeata/gipeda/blob/master/src/BenchmarkSettings.hs#L24 Settings in https://github.com/nomeata/gipeda/blob/master/src/BenchmarkSettings.hs#L85

Am I missing any?

[christetreault]

It seems to me that the format of the file is:

a header (Settings) followed by multiple BenchSettings.

For the Settings:

• title:
  • I assume the only effect of title is to set the page <title> tag?
• diffLink
  • Must diffLink always be the address of the github commit diff?
  • Are the names in {{rev}} and {{base}} important, or will accept whatever I put there?
• revisionInfo
  • Must revisionInfo be an <a> tag?
  • Is the name in {{rev}} important, or will it accept whatever I put there?
• limitRecent
  • I assume limitRecent results in the generated page only showing the X most recent hashes?
• start
  • I assume start hides all hashes prior to the given hash?
• interestingTags
  • What effect does this have?
• InterestingBranches
  • What effect does this have?

for the BenchSettings:

• match
  • How does this work? What does it match the given string against? What is the result?
• smallerIsBetter
  • I assume this makes small numbers show green on the page
• unit
  • I assume this can be any arbitrary unit
• numberType
  • Seems self explanatory. I assume this is used internally to convert strings to numbers
• group
  • What effect does this have?
• threshold
  • What effect does this have?
  • Does this have some upper/lower bounds?
• important
  • What effect does this have?

P.S. After we get this sorted, I can add this in the form of haddocks for Settings and BenchSettings (and maybe readSettings) if you like.

[nomeata]

Hmm, sorry for not documenting this very well. I hope the documentation will grow and improve as there are actually users besides me :-)

Let me explain the settings here for now, and maybe move it to a proper location (more likely README than the haddocs, as this is important for users, while the haddocks are useful for developers). later. If you want to contribute that, that’s of course most welcome!

The settings.yaml supports these keys:

• title:
  Sets the site title.
• diffLink:
  A link that (ideally) displays the difference betwen two commits of the repository. The strings {{rev}} and {{base}} are replaced by the full SHA hashes of the two commits to compare.
• revisionInfo
  Arbitrary HTML code that will displayed on a single commit’s page, useful to reference other resources such as a repository browser or a code review program. The string {{rev}} is replaced by the full SHA hash of the commit.
• limitRecent
  Number of commits shown on the start page.
• start
  Commit hash of the first commit to take into account. Useful to limit the scope of gipeda in projects with a large history.
• interestingTags
  A glob (as understood by git tag -l) specifying which tags should be shown on the main page. By default, no tags are shown; use * to show all.
• InterestingBranches
  likewise, for branches.
• benchmarks: A list of of benchmark settings, explained below:

A benchmark setting supports these keys:

• match
  A file glob that determines to what benchmarks these settings apply: match: "*" will apply these settings to all benchmarks, match: "regression/*"* only to those whose title starts with regression/.
• smallerIsBetter
  By default (or if this is false), gipeda assumes that greater number indicate improvement, and smaller number indicate regressions (e.g. requests per second). This inverts this logic (e.g. runtime)
• unit
  Arbitrary unit, to be printed in the tables and on the graph axes.
• type
  Can be one of float , integral or small integral. For the first two, differences are expressed in percentages (+10%), while for the latter, differences are expressed in differences (+2). This is more suitable for numbers like test suite failures, which should usually be zero or very small.
• group
  The benchmarks are displayed in groups; all benchmarks that have the same string here are grouped under this title.
• threshold
  Any change by a percentage greater than this number is flagged as a regression resp. improvement; anything below this number is considered to be a boring result, and not highlighted separately.
• important
true by default. If false, a regression in this benchmark will not be included in the summary for the whole commit. Use this if you have very volatile tests that would produce too much noise on the main page.

[christetreault]

Excellent, thank you!

Yeah, I suppose the README would be the ideal place to put this stuff. I had been reading the code all morning, and just got "haddocks!" stuck in my head.

I'll leave this issue open for now, until the docs get updated or you get tired of looking at it.

[christetreault]

(I don't want to clutter up your bug tracker, so I'll ask the question here)

Might it be useful to add a BooleanNT to data NumberType?

Our test suite produces a .csv of the form [name],[text execution time],[test successful?], and we're currently just cutting off the success field in our log2csv. I think it might be useful to instead let the [test successful?] field be a separate benchmark.

I could just have that field be a small integer, and 0=false/1=true but I think it'd be cleaner to have a boolean. Additionally, it'd allow you to change how boolean values are represented in graphs and such if you cared to do such a thing in the future.

[nomeata]

Don't be shy opening new tickets, better than long tickets with unrelated discussions.

Your suggestion is quite reasonable, I might add it when I come to it. My only worry would be that it would not be clear what lessIsBetter should mean. Maybe it shouldn't be boolean but rather have good and bad as values.

[christetreault]

Very well, I've opened a ticket: https://github.com/nomeata/gipeda/issues/26