Test::More doesn't prepare test env for script's children

[Astara]

When run under a test harness, I'd expect it to set the ENV vars needed for the test and it's children to run properly, but this isn't the case.

I have a test script "P.t", that Test::More uses to test P.pm. P.t calls P.pm as a program multiple times to get test output and compares the output against "correct responses".

Example of the problem P.pm "used" module "mem". The prereqs were loaded, and the script detected the presence of the mem in the @INC path -- HOWEVER, the @INC path doesn't represent what is in the environment in the PERL5LIB pat. So the child gets a complete different include path than the script program sees. As a result, when it tries to include pre-req modules, it fails.

Test::More needs to set the environment with any options changed in PATH and INCLUDE so they are propagated to children in a test program. Without that, the child runs in some "random environment" that causes unexpected and cryptic failures.... like:

PERL_DL_NONLAZY=1 /home/sand/src/perl/repoperls/installed-perls/perl/v5.12.5/9980/bin/perl "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef _Test::Harness::Switches; testharness(0, 'blib/lib', 'blib/arch')" t/.t Can't locate mem.pm in @INC (you may need to install the mem module) (@INC contains: [...]:.) at lib/P.pm line 150. BEGIN failed--compilation aborted at lib/P.pm line 150.

and

------/PREREQUISITES/------ Prerequisite modules loaded: requires: Module Need Have

---

mem                 v0.3.2 0.3.2  
Test::Simple        0.44   1.001002

build_requires: Module Need Have

---

mem v0.3.2 0.3.2

Where mem is not found, but is theoretically loaded.

[Leont]

It's not Test::More's task to set any environmental variables. Test::Harness sometimes does, but in this case I'd suspect it's your shell script that's being wrong. The command perl isn't guaranteed to be the correct perl installation, and overriding $PERL5LIB that way is also asking for trouble. And exec isn't being passed argument correctly either.

Not to mention the question why does that shell script exist in the first place?

[Astara]

Ah... my mistake, I thought Test::More was a test harness. I'm used to test harness's that set up the test env, run a test script that runs a program with inputs and checks the outputs. But the the script just takes care of running the test, while a harness takes care of setting up the env... so, sorry, I thought Test::More was supposed to be a test harness.

The shell script is the test-harness that sets up the environment so the tests can run correctly. AT least thats the idea, as you say -- since you've said that it's not Test::More's responsibility. Something has to set it before perl is run, and if it isn't the harness, is a script. That explain how it is?

[karenetheridge]

Something has to set it before perl is run, and if it isn't the harness, is a script.

Your test (the .t file) itself should be responsible for setting up the environment needed for any subsequent tests to properly run.

[Astara]

On 11/11/2013 11:09 AM, Karen Etheridge wrote:

Something has to set it before perl is run, and if it isn't the
harness, is a script.

Your test (the .t file) itself should be responsible for setting up the environment needed for any subsequent tests to properly run.

— Reply to this email directly or view it on GitHub https://github.com/schwern/test-more/issues/390#issuecomment-28228911.

It is not able to, according to the documentation in Test::More.

I had two problems that I needed to solve -- one was correct setting of the PATH and PERL5LIB so that children would work correctly, and two was deleting and restarting, if necessary, if there were corrupting values in PERL5OPT (-Mutf8 creates corrupt output even though the tests contain utf8 and say "use utf8;" in the source, utf8 strings in the source are corrupted and either treated as bytes OR upgraded again!

I couldn't get them to just be left as pre-encoded utf8 strings.

I was told that I could not unset PERL5OPT and undo it's effects -- that they had to be unset before the program was run (by some outside program -- that's why the shell script is there).

If you know of a better way, I'm open, but I have PERL5OPT="-Mutf8 -CSA" in my env -- needed for my system's programs to run reliably. If I unset that programs on my system stop running. So I need for the test cases to unset that value before running tests...

[karenetheridge]

Your test (the .t file) itself should be responsible

It is not able to, according to the documentation in Test::More.

I didn't say Test::More, I said your test. Your test is perl and can manipulate the environment. If you intend to call out to an external script, you should set up whatever preconditions that script requires.

All that Test::More does is provide a mechanism for comparing actual results against expected ones. It knows nothing about the code being tested or how it is supposed to work. That's on you, the test writer.

I would strongly recommend re-reading https://metacpan.org/pod/Test::Tutorial.

[Astara]

On 11/11/2013 2:54 PM, Karen Etheridge wrote:

    Your test (the .t file) itself should be responsible

It is not able to, according to the documentation in Test::More.

I didn't say Test::More, I said your test. Your test is perl and can manipulate the environment. If you intend to call out to an external script, you should set up whatever preconditions that script requires.

All that Test::More does is provide a mechanism for comparing actual results against expected ones. It knows nothing about the code being tested or how it is supposed to work. That's on you, the test writer.

I would strongly recommend re-reading https://metacpan.org/pod/Test::Tutorial.

---

I read it again, but it doesn't tell me anything pertinent to this.

There are 2 programs -- one is the program running the test, and the other is the program being tested.

So what do you mean when you say "your test"? The program being tested, or the script that drives the program?

P.t uses P.pm for its own output, but invokes P.pm (as a program), passing it various inputs for testing. So P.pm is a program being tested AND an included module to generate output.

When it invokes P, one thing that can go wrong is that if, whatever invokes P.t, may do so with a custom perl in /usr/local/perls/perl-47/bin/perl which is not in the PATH.

So when P.t calls P.pm, P.pm will be run with the perl found via PATH.
(lets say in /usr/bin). It defaults to using libs in /usr/lib/perl5/V.xx/...

So now P.pm is running with a tree where it's prereqs are not installed and with the wrong version of perl...

P.t, could set path, and lib, but it CAN'T change the options PERL5OPT that it was invoked with. So if PERL5OPT had "-d" in it, P.t would enter the debugger before it ever got control. That's what happens when PERL5OPT has -Mutf8 -CSA in it -- They are executed before P.t ever gets control, so it cannot change them for itself.

It's Test::More that can't read P.pm correctly.

The page you pointed me at doesn't show 1 example of calling a program and checking its output. It shows checking results of calls to functions in the same program (P.t), which isn't what my test is doing.

Am I making more sense?

[karenetheridge]

"Your test" is your .t file.

If you invoke a separate process and want to test its output, you need to first of all invoke the process properly (that's up to you to control), and then capture stdout/stderr into a string. Then you can use Test::More to compare the actual output from what you expected.

None of this needs to be tied to a Test::* module - you are free to invoke your process any way you like, and capture its streams any way you like. Capture::Tiny and Test::Output are two popular approaches here, but there are more as well.

It's Test::More that can't read P.pm correctly.

That statement makes absolutely no sense. It's not trying to read anything; all it does (assuming we're talking about the is function) is compare two strings for equality, and emit the proper TAP output indicating the result.

You seem to be very confused about the differences between a module and a program, how to invoke separate programs, how to capture output, and what the Test::Builder framework actually does.

For that matter, I see no reason why you need to invoke separate processes at all to test a module that seems to be all about turning data structures into strings. Why not do everything in the single process?

[Astara]

On 11/11/2013 4:43 PM, Karen Etheridge wrote:

"Your test" is your .t file.

If you invoke a separate process and want to test its output, you need to first of all invoke the process properly (that's up to you to control), and then capture stdout/stderr into a string. Then you can use Test::More to compare the actual output from what you expected.

---

The above is what I do. It invokes the program via 'perl' using one of the two formats (with or w/o an extra param). "$tp"= P.pm, which has the output I am testing.

It is a module that is normally used via "use P". the self-demo/test output is in a DATA section that is "eval'ed" if "P.pm" is invoked as a program.

The extra param is to allow output redirection to test that printing to some other file handle than "STDOUT" works correctly.

sub getcase($;$) { my $case = shift; my $cmd = @? "perl $tp $case ".$_[0]."|" : "perl $tp $case |"; open(my $fh, $cmd) || return undef; my $out; { local $/=undef; $out = <$fh>; } chomp $out;

$out;

Then it slurps the output & returns it. This was working until I added the mixed numeric test... (in response to a bug ).

The source, itself, is already utf8. The -CSA say that STDIO is already UTF8 encoded.

When it runs, the last example shows: #13 (test mixed digit string): embed roman pi = ["3.ⅰⅳⅰⅴⅸ"] which is basically a test for

Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=89063 >

None of this needs to be tied to a Test::* module - you are free to invoke your process any way you like, and capture its streams any way you like. Capture::Tiny and Test::Output are two popular approaches here, but there are more as well.

It's Test::More that can't read P.pm correctly.

That statement makes absolutely no sense. It's not trying to read anything;

---

Um...you're right, it assumes the output is Latin1. Trouble is, the output is already UTF-8 encoded, so instead of UTF-8 characters, it printed them as latin1 bytes even when I had "PERL5OPTS=-Mutf8 -CSA" (source in utf8, STDIO+args in UTF8):

not ok 29 - test mixed digit string

Failed test 'test mixed digit string' at t/np.t line 129. Regex

"(?^u:embed.ⅴⅸ.$)" did not match string "embed roman pi

= ["3.ⅰⅳⅰⅴⅸ"]" in case "test mixed digit string".

Noticed the pattern DID get treated as utf8 -- it's in a BEGIN statement, but the string did not. So I tried adding:

BEGIN { my $builder = Test::More->builder; binmode $builder->output, ":utf8"; binmode $builder->failure_output, ":utf8"; binmode $builder->todo_output, ":utf8"; }

as suggested on the manpage. That gave same thing. So without Test::More, it prints correctly, but with it, it always fails the match.

(presence of "-CSA" makes no difference -- only removing "-Mutf8" from my PERL5OPTS allows the possibility of it matching. P.pm requires "use utf8" as it has utf8 in it. So does P.t -- bug the test program (with T::M included) cannot have "use utf8" even though it has utf8 in it. (which seems like a bug "somewhere"??....

So without using the recommended binmode stuff on T::M->builder(output/failure_output/todo_output), it works.

If I do use it, another test fails that uses a unicode ellipsis.

(i.e. WITH: BEGIN { my $builder = Test::More->builder; binmode $builder->output, ":utf8"; binmode $builder->failure_output, ":utf8"; binmode $builder->todo_output, ":utf8"; } I get: not ok 25 - P @{[FH,["fmt:%s",…]]}

Failed test 'P @{[FH,["fmt:%s",…]]}'

at t/np.t line 126.

Regex "(?^:fmt:Hello Perl,)" did not match string "fmt:Hello Perl 11"

in case "P @{[FH,["fmt:%s",…]]}".

---

It turns out the above test would fail with the "," after Perl no matter what! I can't come up w/a case that works w/T::M that works for both 25 & 29 (25 should look like:)

ok 25 - P @{[FH,["fmt:%s",…]]}

all it does (assuming we're talking about the |is| function) is compare two strings for equality, and emit the proper TAP output indicating the result.

Well, yeah -- it's likely the Test::Simple that it is based on -- but really, the above symptoms have me very confused.

Please note, though, if I am confused it's ALSO because this is my 1st CPAN module!!!...so I do not claim any expertise in the perl testing setup. (I have had QA experience automating testing bootup and BIOS output on diverse PC's to enable installing specific SW on bare-bones PC's (i.e. not only no perl available on what was being tested -- no OS either! ;-)).

Anyway, that's why I tried the shell script.

But I didn't know the test was hard coded to run as a perl script. so need a perl version of that script and that might pass testing, but I know the tests would fail if they were more rigorous -- not due to any fault in P (it is printing the right output), but due to use of the harness stuff which is messing up how utf8 is interpreted.

---

Well at some point

You seem to be very confused about the differences between a module and a program, how to invoke separate programs, how to capture output, and what the Test::Builder framework actually does.

---

If you saw the above symptoms, might you not be confused?

For that matter, I see no reason why you need to invoke separate processes at all to test a module that seems to be all about turning data structures into strings. Why not do everything in the single process?

---

How do I capture STDOUT or STDERR? P does different things depending on if it is printing to a string or to output

I.e. I need to run it and intercept STDOUT and later STDERR to see if the correct FH was written to. Writing to a string is only 1 thing it does -- (ooo... just thought of another test case I should add)

When printing to output, P generates different output depending on the value of "$\", so if $\ is set to newline, P won't generate an extra one.

How it handles automatically printing to a file handle or a string or automatically changing handling based on $\ or it's newline suppression (0x83) at the end of a string can't be tested except by printing to a file handle.

I would welcome you trying it -- especially if you change the "match" pattern for case 25

—

from : push @answers, [11, qr{fmt:Hello Perl}]; #",…" - causes trouble

to:

push @answers, [11, qr{fmt:Hello Perl,…}];  #",…" - causes trouble

With that change, seems any set of options still results in either 25 or 29 failing.

The entire run of "P.pm", as an executable, looks like:

1 (ret from func) : Hello Perl 1

2 (w/string) : Hello Perl 2

3 (passed array) : Hello Perl 3

4 (w/fmt+string) : Hello Perl 4

5 (to STDERR) : Hello Perl 5

6 (to strng embedded in #7):

7 (prev string) : prev str="Hello Perl 6" (no LF) && Hello

Perl 7

8 (P && array ref) : ["one", "two", "three", 4, 5, 6]

9 (P HASH ref) : {a=>"apple", b=>"bread", c=>"cherry"}

10 (P Pkg ref) : Pkg{a=>1, b=>2, x=>'y'}

11 (P @{[FH,["fmt:%s",…]]}) : fmt:Hello Perl 11

12 (truncate embedded float): norm=3.14159265358979, embed={pi=>3.14}

13 (test mixed digit string): embed roman pi = ["3.ⅰⅳⅰⅴⅸ"]

Roughly half of the tests are checking to see if it received the output from the child, with the other half being regex checks... so it's not real long...

Hope that better explains what I think I understand as well as what is going on.

Cheers, Linda

[shadowcat-mst]

A sane solution to the underlying problem would be to take the following code in P.pm -

package main;

use 5.8.0; use utf8;

unless ((caller 0)[0]) {
$_=do{ $/=undef, <P::DATA>};
    close P::DATA;
    our @globals;
    eval $_;
die "self-test failed: $@" if $@;
    1;
} else {
    close P::DATA;
}

do the slurp of into a variable no matter what happens, and do expose the script running thing as a subroutine, maybe run_as_script.

Then you can write

unless ((caller 0)[0]) { run_as_script() }

to preserve the "run as a script" behaviour but you can call P::run_as_script() directly from your test code.

At which point you can throw out the subprocess overcomplication, delete that shell script, and worry about how to spawn perl processes from test files later when you've got a genuine case where you need to understand that.

THis still isn't a Test::More bug but good luck refactoring P.pm so it can be tested sensibly!

[Astara]

On 2013-11-12 09:06, shadowcat-mst wrote:

At which point you can throw out the subprocess overcomplication, delete that shell script, and worry about how to spawn perl processes from test files later when you've got a genuine case where you need to understand that.

---

I understand what you are saying, BUT, the test driver program has to be able to read both STDERR and STDOUT of the child program.

P writes to STDOUT by default. Pe writes to STDERR by default. however, if you "capture" the output via $var=P(...). Then P doesn't write to STDOUT by default.

If you explicitly specify a file descriptor (or use Pe) AND capture the output, then the output is printed to the FD AND returned as it's value.

I.e. a test program can't simply "capture" the return value of P. The test program needs to connect STDIN to STDOUT (something that happens automatically if P is run as a child), OR, if I am testing Pe (or print to a FD), I need to connect STDIN to STDERR or the FD (something that doesn't happen automatically, but isn't that difficult -- must save& close STDOUT, then dup STDERR to STDOUT before the fork and restore afterwards.

I don't think you understand -- I need to read the file handle(s) that are STDOUT and STDERR of "P". If I call P as a subroutine, and capture output, I don't get output over the file handles _AND_ the output is different, as it goes out to a file handle vs. how it is returned from P when called as a sub.

The main function is to replace doing I/O to file handles, not as an sprintf replacement. What you suggest doesn't allow testing I/O -- only the "sprintf" function which is a relatively minor portion of "P".

The bug may not be in Test::More -- but in whatever harness is calling P.t -- which isn't assuming utf8 encoding when it should. (it ignores the values in PERL5OPT).

[karenetheridge]

On Tue, Nov 12, 2013 at 11:44:07AM -0800, Astara wrote:

I.e. a test program can't simply "capture" the return value of P.

You can capture stdout and stderr. I gave you two popular modules that do so.

I don't think you understand -- I need to read the file handle(s) that are STDOUT and STDERR of "P".

Okay, so read the filehandles! That's not hard.

The main function is to replace doing I/O to file handles, not as an sprintf replacement. What you suggest doesn't allow testing I/O -- only the "sprintf" function which is a relatively minor portion of "P".

If you structure your program in such a way that it's impossible to test, you shouldn't blame that on the testing infrastructure, but rather your code. There's no reason why you can't capture stdout and stderr from an external program and then verify its contents.

The bug may not be in Test::More -- but in whatever harness is calling P.t -- which isn't assuming utf8 encoding when it should. (it ignores the values in PERL5OPT).

There may not be a test harness at all. Does your code work if you call it directly, that is via /some/path/to/perl -Ilib t/foo.t ? Get that working first, and then everything else will work on top.

It's up to you whether or not to ignore all the advice you've been given on this and prior threads. I won't be responding to this ticket again.

[shadowcat-mst]

"I understand what you are saying, BUT ... "

I tried to give you some hints while explaining this isn't a Test::More bug

So long as you're now clear this isn't, @schwern can close it, and you can redirect your support request to another, more appropriate venue. You might want to try perlmonks; they're usually quite sympathetic to newbies trying to learn the basics and certainly more so than I'm capable of.

[Astara]

First, let me re-iterate:

The bug may not be in Test::More -- but in whatever harness is calling P.t -- which isn't assuming utf8 encoding when it should. (it ignores the values in PERL5OPT).

---

I'm willing to believe the problem isn't exactly in Test::More.

Here's the weirdness. This module and tests worked UNTIL, I added test "29" -- which had UTF-8" encoding in it.

So all the bit about spawning a child, and reading test handles... that's all handled and worked.

What isn't working (and may have nothing to do with Test::More)

is "unicode!" in the test -- when it "runs" and displays to STDOUT, it displays correctly, and I don't get any "wide character in output" errors. So utf8 settings are correct for "stand alone".

****>>>If I don't have PERL5OPTS set with -Mutf8 in it, the test works "fine". My grand solution was to run a perl5 script that unsets that and then calls the test.

What I don't understand (and no one on perlmonks - so I tried there), was what was causing it to fail if PERL5OPTS was set with "Mutf8" in it, and how to compensate for it in the P.t program rather than having a wrapper "P.env" unset PERL5OPTS and then call "perl P.t"...

I don't like the wrapper solution, but it works. Feel free to close this out as an issue against Test::More at this time.

If I feel motivated enough to figure out why it's not working and why the already-utf8 encoded string is being interpreted as latin1, and if I come to the same conclusion (problem in Test::More)-- I can either re-open or refile. But I'm more likely to believe it is something about the program that forms the test code being read from an embedded data section AND utf8 is in PERL5OPTS... I think that maybe the utf8 isn't being applied to the DATA part of the program..

So please understand -- I'm 1 step above "clueless", in that it works w/o unicode! ;-) It was adding that line of unicode that has given me headaches.... But I need that line as it represents a bug case. (but that's my issue)...

Copacetic?

Linda

[rjbs]

Tests are generally run by Test::Harness or TAP::Harness, neither of which is part of Test::More.

I'm closing this ticket. TAP-Harness has its own issues list.