search

Raku / problem-solving

🦋 Problem Solving, a repo for handling problems that require review, deliberation and possibly debate
Artistic License 2.0
70 stars 16 forks source link

perl6-infra: Host the Blin Toaster #32

Open rafaelschipiura opened 5 years ago

rafaelschipiura commented 5 years ago

https://github.com/perl6/Blin

I suggest hosting the results and making it toast automatically.

AlexDaniel commented 5 years ago

Toaster is superseded by Blin (or at least I think so), and the data generated by Blin is used to automatically populate the issue tracker in the ecosystem-unbitrot repo. Last update is Feb 27 because I was extremely busy with stuff, but I'll try to run it more regularly from now on and maybe we can even automate it.

AlexDaniel commented 5 years ago

To clarify, Blin and ecosystem-unbitrot are hostless. Anyone can run Blin to look at the results locally, and there's a script in ecosystem-unbitrot that takes the file from Blin and updates the tickets. I think this is pretty cool, but people keep requesting a subdomain with results… 🤷

rba commented 5 years ago

Ok. I learnd that this as to be run in separated environment. And there is some (huge) cpu power needed? How often should this run or how is it triggered?

AlexDaniel commented 5 years ago

@rba usually the release manager (recently me and nowadays @kawaii) will run it a few times before making a release (so a few times during a month). It doesn't take too much, I used google compute engine with 24 cores and it takes only 1 hour or so. How often it should be run depends on the state of rakudo… for example, if some breaking changes land, then there's usually a period when things are expected to be broken a little bit, so Blin can help a lot to find all the remaining bugs but ecosystem-unbitrot should not be populated with that data (or so I think, because it will ping module authors needlessly for issues that are out of their control).

If we have a server that we can abuse like that every day for free, then maybe indeed we can automate it.

kawaii commented 5 years ago

I've been running Blin recently on a server provided for R&D purposes by $dayjob, although I imagine that isn't an arrangement that will last indefinitely. It might be worth reaching out to somewhere that can sponsor such a server if we'd like to run it daily on a continuous basis.

rafaelschipiura commented 5 years ago

@rba: @AlexDaniel is talking about one of the uses of the software, which is to test Rakudo before release. But I think it's useful also for module authors if they can have this information at hand, if they can expect trouble because of a release.

So that's why I think it's nice to have it somewhere. Once a day would be plenty, no need to run it on every HEAD commit.

AlexDaniel commented 5 years ago

But I think it's useful also for module authors

Not sure if I'm succeeding at conveying the point :)

When I was the release manager, I used the data generated by Blin to populate https://github.com/perl6/ecosystem-unbitrot/issues. Tickets there include a @… mention of the author (when available), so module authors (at least those affected) had the information at hand…

rafaelschipiura commented 5 years ago

Right, I think it's useful if module authors can pull that information instead of waiting for a push.

This way they can know if everything is alright and not the release manager forgetting it.

But yes, your suggestion makes it not critical. Is it documented in a release guide somewhere?

AlexDaniel commented 5 years ago

Moreover, for every release my goal was to have 0 modules affected, which was mostly the case once all regressions were fixed before actual releases were made. Of course, some modules were just wrong in one way or another, but in these cases I'd contact the author myself (by at least filing a ticket, although typically I'd try to submit a proper PR that resolves the issue). The process was usually reflected in tickets like these: https://github.com/rakudo/rakudo/issues/2705 https://github.com/rakudo/rakudo/issues/2638 https://github.com/rakudo/rakudo/issues/2451

Toaster is briefly mentioned in https://github.com/rakudo/rakudo/blob/master/docs/release_guide.pod, but Blin is not (it was a more recent addition). Also, it seems like the release guide does not stress the “0 modules broken” goal, even though it was the case for at least 17 releases that I did.

@kawaii maybe you should tweak the release guide accordingly?

Also, maybe we should mention somewhere in the docs that we do that. Like in https://docs.perl6.org/language/modules-extra or some other page. @JJ, any thoughts?

jnthn commented 5 years ago

Tying in to issue #5: one practical issue when we get pull requests is knowing the fallout, and I think we'd feel more confident merging things if we had confidence they would have relatively little fallout.

I believe Travis so far only runs make test, not even make spectest. But the ideal would be to not only have it spectested, but also have something that applies the PR, runs it against the ecosystem with Blin, and posts the results vs. master.

I guess that's easier to put in place if we have BaaS (Blin as a Service). :-)