search

thaljef / Pinto

Curate your own repository of Perl modules
https://metacpan.org/module/Pinto::Manual
66 stars 49 forks source link

Avoid calling cpanm in test suite when NO_NETWORK_TESTING=1 is set #170

Closed karenetheridge closed 9 years ago

karenetheridge commented 9 years ago

I just discovered that the low number of cpantesters reports for Task::Kensho::Toolchain might be due to Pinto calling into cpanm (at least 20 times, it looks like!) during its tests Every call into cpanm creates a new ~/.cpanm/*/build.log, which interferes with cpanm-reporter's ability to find the proper build.log and submit it to the testers network after installation.

Please, could you refrain from running tests that touch the network (including invoking cpanm) when the NO_NETWORK_TESTING environment variable is set?

thaljef commented 9 years ago

NO_NETWORK_TESTING

I didn't know such a thing existed. Will do :)

thaljef commented 9 years ago

Putting aside the issue with ~/.cpanm/build.log, does NO_NETWORK_TESTING mean that I shouldn't create any kind of TCP connection, even if they only connecting to localhost?

thaljef commented 9 years ago

NO_NETWORK_TESTING doesn't seem like the right choice here, since all connections are to localhost, so it's not really network testing. Maybe the PERL5_CPAN_IS_RUNNING family of variables would be better. Or is there some other variable that is usually present for CPAN Testers?

karenetheridge commented 9 years ago

On Sat, Nov 01, 2014 at 04:00:42PM -0700, Jeffrey Ryan Thalhammer wrote:

NO_NETWORK_TESTING doesn't seem like the right choice here, since all connections are to localhost, so it's not really network testing. Maybe the PERL5_CPAN_IS_RUNNING family of variables would be better. Or is there some other variable that is usually present for CPAN Testers?

There isn't.

Let's get to the heart of the matter: I find it objectionable that installing Pinto causes the generation of several invalid (that is, not generated by me the user) ~/.cpan/work/xxx/build.log files, as it interferes with my ability to send test reports. Tests shouldn't be writing, directly, or indirectly, to files outside its own directory or $TMPDIR.

Would you consider moving the cpanm testing to xt/, or finding some other way of testing Pinto that doesn't directly invoke a real cpanm instance?

thaljef commented 9 years ago

I find it objectionable that installing Pinto causes the generation of several invalid ... files.

No argument here. I'm just questioning whether NO_NETWORK_TESTING is the most appropriate hook. And if there isn't a better one, then your xt/ suggestion would be a good alternative.

thaljef commented 9 years ago

Starting in 0.09999, there are no (user) tests that require a network. Everything goes to localhost and it will do-the-right-thing even if you've set a proxy. Also the by-products of running cpanm during the tests are isolated and discarded afterwards.