the big idea

[rjbs]

CPAN-Reporter-Local

Here's the plan:

• something (a person or program) sets up a test box, probably a Debian image running on Linode

• it installs the basics needed to make it capable of doing smoking

• for each experiment, we need:

  • a control, some stable build of perl whose CPAN test results we collect
  • the experimental build, a version of perl with some change to config or code

• we test the control build n times

• we test the experimental build n times

• testing is done with CPAN::Reporter::Smoker

• CPAN::Reporter::Smoker will automate CPAN to run all the tests we want

• maybe we should set the CPAN config trust_test_report_history to avoid re-testing the same thing repeatedly

• really there are a bunch of options described in CPAN::Reporter::Smoker to consider

• we configure Test::Reporter to use Test::Reporter::Transport::File at least for now; this writes results to disk, does not send them to CPAN Testers

  • CPAN::Test::Reporter will use the env var $PERL_CPAN_REPORTER_DIR as its home for settings; see below for example contents

• when testing is complete, we run a comparison to find things that reliably pass on control but fail on experimental

• we need a list of distributions to test

CPAN::Test::Reporter config

Example:

edit_report=no
email_from=rjbs@cpan.org
send_report=yes
transport=File /tmp/smoke-runs/BUILD-ID/test-reports

[rjbs]

I spent some time digging into how we'd run two testers on one box. It's probably not a big deal, if we put our minds to it, but it looks a little annoying, because you can't tell CPAN that its config lives in ~/.cpan-xyz instead of ~/.cpan.

So maybe the simpler thing is to make one test machine (or container) per version. We'd end up with the reports on two different machines, and then copy them off for analysis, or maybe produce a compact summary: list of results plus copies of only failures?

[rjbs]

Setting up a minicpan is a little slow. We should try to an image of one available to attach immediately.