search

sgraf812 / feed-gipeda

Daemon for watching for new commits to multiple git repositories and feeding benchmark runs into gipeda
BSD 3-Clause "New" or "Revised" License
5 stars 1 forks source link

How do you handle build failure? #5

Closed nomeata closed 8 years ago

nomeata commented 8 years ago

Looking at the README (and a bit at the code), I still wonder: How do you handle build failures?

Often, commits in a repository are broken, and do not even build. For these, no .csv file should be fed to gipeda. But also, these should not be re-built on every invocation (or even iteration) of feed-gipeda.

sgraf812 commented 8 years ago

Currently, build failures aren't handled in any specific way I'm afraid.

Can gipeda handle an empty CSV file? Otherwise I have to invent some other mechanism to check for already benchmarked repositories. The idea is to generate an empty CSV file in case of build failure.

nomeata commented 8 years ago

The idea is to generate an empty CSV file in case of build failure.

Quite contrary: gipeda expects no CSV in case of an build failure, and will then skip this commit when showing graphs and tables, and show useful improvement/regression markers. With an empty CSV, it would assume that the build succeeded and the test case was missing, so it would not consider any benchmarks as regressed in the next commit.

Eventually, I’d like to improve gipeda to discriminate between commits not yet built, and commits that failed to build, and take that into account. The question is how best to communicate that to gipeda. Maybe an empty CSV file is the right indicator? Maybe a bit too implicit... Alternatively, a CSV file that says failed_build;1? But that’s also a bit ugly.

Suggestions?

sgraf812 commented 8 years ago

Maybe a file <SHA>.failed? Or an empty CSV file under site/out/failures.

Otherwise, something like a benchmark section build/errors in addition to build/warnings might not be that bad after all, although that would mean to hardcode for a convention.

nomeata commented 8 years ago

Maybe a file .failed? Or an empty CSV file under site/out/failures.

The former is actually what I do right now (in the logs directory), with the filename convention <SHA1>.log.broken: https://github.com/nomeata/ghc-speed-logs/commit/b37e5672ec2478321102644522b6f58c433f202c

This way, gipeda can link to the failed build log, which is likely very useful when debugging these issues.

But anyways, that is future work in gipeda at the moment. For now, you have to make feed-gipeda to avoid passing an empty CSV file to gipeda, while at the same time not building it twice.

sgraf812 commented 8 years ago

I thought a little more about which mechanism I should employ:

  1. Transparently create a commits.txt on the same level as repository/ and site/ where I save all handled commit SHAs
  2. Employ the same log/ mechanism as you did. That would entail some more work, because I have to separate the responsibilities of cloben, which currently doesn't output logs.

Question is: How valuable are the logs? They can't be accessed from the web service anyway.

nomeata commented 8 years ago

They can't be accessed from the web service anyway.

Sure, there is a “view buildlog” link from https://perf.haskell.org/ghc/#revision/81b14c141dc385dbb0de00ea72217185cfa22a09 and it has occasionally proven useful.

What is your current algorithm to enumerate what commits to build? How do you handle tags, branches, and how does feed-gipeda know what the oldest commit is it should build? (Building everything is not a good idea if you have a repository like ghc’s with ancient commits).

Gipeda has a setting that sets the first commit to look at, as well as patterns to match “interesting” branches and tags.

Given that it is slightly non-trivial: maybe this functionality should move to gipeda? I could imagine gipeda producing a list of SHA1 hashes that it is missing data for.

sgraf812 commented 8 years ago

Sure, there is a “view buildlog” link

Oh, must have overseen this. I should really play around with it more. How do you generate the links? I'm a little worried that writing to the logs/ directory will not be sufficient and that I have to update another GitHub repo while I am at it.

current algorithm to enumerate what commits to build

I'm basically parsing the output of git log --format=format:%H, which I think spits out all commits leading to HEAD (defaulting to master, probably).

maybe this functionality should move to gipeda?

It would certainly be easier to have the functionality in one place. Meanwhile I could modify cloben in a way that would allow it to generate the log files.

nomeata commented 8 years ago

How do you generate the links?

They are configured in the gipeda configuration (revisionInfo): https://github.com/nomeata/gipeda/blob/master/ghc/settings.yaml

Of course they are optional, and can be adjusted to whatever hosting layout one uses.

I'm basically parsing the output of git log --format=format:%H, which I think spits out all commits leading to HEAD (defaulting to master, probably).

That will not work well with repositories with a long history. The best idea I had was to require people to indicate a first revision; see the above settings again.

It would certainly be easier to have the functionality in one place. Meanwhile I could modify cloben in a way that would allow it to generate the log files.

Log files are not a crucial feature; but detection of build failures are.

Maybe we should discuss this in person. Feel free to come by my office (today between 12:45 and 14:30, or tomorrow).

sgraf812 commented 8 years ago

Log files are not a crucial feature; but detection of build failures are.

Then I'll go for the approach 1. above for the time being.

Feel free to come by my office

OK, will stop by some time between 13:00 and 14:00.

sgraf812 commented 8 years ago

The current version watches for changes to a backlog.txt containing unhandled commits (see the current README's 2nd image). Currently, I produce it myself through just listing all commits of a repository.

That's clearly not the intended behavior (given gipeda settings such as start commit), but IIRC, this text file will ultimately produced by gipeda itself, at which point I can just delete that part from feed-gipeda.

So, this should pretty much work in the future.