search

noxxi / p5-io-socket-ssl

IO::Socket::SSL Perl Module
36 stars 62 forks source link

Test suite fails on Windows #161

Closed eserte closed 1 week ago

eserte commented 1 week ago

I tried to install the IO::Socket::SSL 2.089 myself within a Win10 VM using an somewhat older Strawberry Perl, but the test suite fails:

t/mitm.t .......................... 
Failed 1/8 subtests 
...
t/psk.t ........................... 
Dubious, test returned 8 (wstat 2048, 0x800)
Failed 8/14 subtests

I do not seem to be the only one with failed tests: http://matrix.cpantesters.org/?dist=IO-Socket-SSL+2.089 shows only red for MSWin32. (Unfortunately cpantesters is currently broken and it is not possible to see the individual test reports at the moment)

Also Strawberry Perl itself seems to skip the tests or at least parts of it: https://github.com/StrawberryPerl/Perl-Dist-Strawberry/blob/8ec82760ac883c7a6a00425fc2ec53967526b528/share/64bit-5.40.0.1.pp#L149 (Here it's not entirely clear if the test suite is completely skipped (skiptest=>1) or only the test files listed in HARNESS_SKIP)

Unfortunately I know nothing about Windows. I deactivated everything which looks like a firewall, but this does not help. The mitm.t test seems to work sometimes if run individually from cmdline, outside of make test.

noxxi commented 1 week ago

Unfortunately tests on Windows are kind of unstable compared with UNIX and MacOS.

This is likely not due to a failure in IO::Socket::SSL itself but on how most tests work. These start several processes (client and server) which then communicate with each other. On UNIX + MacOS this is done with fork which is stable. On Windows platforms there is no real fork though and this is only emulated by Perl which behaves slightly different and can lead to unusual race conditions. In addition to that Antivirus on Windows platforms hook deep into the systems and might interfer with the tests in a bad way if they think they behave suspicious.

All of this makes Windows an unreliable platform for executing the tests. Sometimes it works on many systems but not on others, sometimes one test fails at one system but a different test fails on another system. And mostly non of this is reproducible for me, which means I cannot really fix it.

But, likely the code will work even if some tests might fail.

eserte commented 1 week ago

Just for reference: I am using the following CPAN.pm distroprefs file to skip the test files which are problematic in my setup: https://github.com/eserte/srezic-cpan-distroprefs/blob/5d126243d9784cf63f9d02510144b3d8b3b80254/IO-Socket-SSL.yml#L31-L52