How to view program output?

[pcworld]

For most tests, I currently only see whether they have failed or succeeded. It would be very useful to be able to actually see command inputs ands program outputs. Because right now I still have to create input files and run my program myself to actually see what my program puts out (e.g. I want to figure out why my program prints out an error message even though it shouldn't, and even when tests succeed I would like to see if my error messages make sense).

Is there an easy way to do this? If this isn't possible right now, maybe we could implement a log option which creates a file that for every test contains: the test name, whether the test succeeded, input file contents, and command inputs/outputs. E.g.:

### ValidNodesCommandTest#taskSheetExampleTest (success)
# input file:
CentOS5 ( id= 105) contained-in operatingSystem
centOS6 ( id = 106) contained-in OperatingSystem
operatingSystem contains centos7 ( id = 107 )
operatingsystem contained-in Software
CentOS7 (id=107) successor-of centos6(id=106)
CentOS5 (id=105) predecessor-of centos6(id=106)
writer (id=201) contained-in officesuite
calc (id=202) contained-in officesuite
impress (id=203) contained-in officesuite
officesuite contained-in software
LibreOffice (id=200) contained-in officesuite
writer (id=201) part-of LibreOffice (id=200)
calc (id=202) part-of libreoffice (id=200)
libreoffice (id=200) has-part impress (id=203)
# CLI:
$ nodes
calc:202,centos5:105,centos6:106,centos7:107,impress:203,libreoffice:200,officesuite,operatingsystem,software,writer:201
$ quit
# program exited with status code 0

Of course something easier would suffice too, I just want to be able to see my program output without having to copy&paste all test input files and having to run the program myself for all tests.

[jGleitz]

I feel the need to clarify things. The test output behaviour is not perfect, but also not that bad. I have included images for how it looks in Eclipse on Windows. I think this is the current situation:

• If an exception is thrown by the tested class, the user sees

  • An error description
  • A stack trace showing him were the error occurred

  meanwhile, he does not see

  • the input file
  • the commands

• If the tested class produces wrong output or uses the wrong exit status, the user sees

  • An error description
  • A stack trace showing him were the error occurred
  • the commands
  • the tested class' output

  in the moment, he does unfortunately not see

  • the input file

[jGleitz]

For now, I always use the stack trace to jump to the test method. That's why I recommend putting input directly there. Because looking at the input file is then only one click away: Click the last item in the stack trace and you see it.

[pcworld]

Yes, but what I e.g. miss is a way for me to see my error messages for tests that expect error messages. If I could do that, I would have less need for launching my main method myself, and could make sure a meaningful error message was output. And for other succeeding tests I could trust the tests more if I could see input and command history etc. (e.g. I haven't actually tested if this for loop does what I think it does, I just believe it with high probability).

Basically, I like the whole test thing and am too lazy to launch my program myself to see how it works :)

[jGleitz]

Ah, now I get your point. Indeed, it would be very nice to see the output of succeeded tests. However, a quick research told me this is not possible with JUnit (correct me if I'm wrong!). We'd have to implement our own logging mechanisms.

[pcworld]

Some real world scenario: My code failed for the InvalidInputFileTest#invalidProductIds "big number" test. I changed something in my code, the test succeeded. However to see if my program now actually outputs the correct error message, I have to manually copy&paste the test input file and launch my program.

[jGleitz]

@pcworld I absolutely see your point. It's a major shortcoming of our framework. Nevertheless I don't know how to solve it. We could use wrappers around the existing JUnit-Matchers to get the output and log it. But I can't think of any good way to represent it!

[pcworld]

We could use wrappers around the existing JUnit-Matchers to get the output and log it.

TestObject already handles all input and output. Though there would probably be an issue of correlating the input/output to a specific test.

But I can't think of any good way to represent it!

Me neither (I don't know anything about JUnit though). I'd say a log file is the way to go. Example file format see in first post.

[jGleitz]

I'd say a log file is the way to go. Example file format see in first post.

I have a strong anticipation towards programs that clutter my disk with log files. If there's a bug in your program, you'll end up having a lot of test runs and tons of log files.

[pcworld]

It should be configurable, yes.

[jGleitz]

Hm… is printing to System.out an option?

[pcworld]

If it ends up in the console, sure.

[jGleitz]

Maybe I'll look into this today. What do you think is the easiest way to implement a debug option? I see two possibilities, but am happy for more ideas:

• make an debug option in the VM arguments (which the user has to set in its run configuration. He could also have two run configurations, a verbose and a silent one)
• Detect when the user's running the test in Eclipse's debugger and print in that case (I read that this is possible. This would make the option faster accessible, but maybe not everybody wants to have his console cluttered when debugging)

What do you think about it?

[pcworld]

First option sounds better to me. That would e.g. easily allow to put the tests in a script that does something with the output.

(Last time I tried JUnit's debug option (to debug an exception) didn't really work for me. The test framework might have killed the process after a timeout. But this issue is rather unrelated to this.)

[jGleitz]

Allright, then we'll go for the first. Regarding debugging: See #17

[jGleitz]

I'd suggest postponing this in favour of getting final task 1 test ready.

[pcworld]

I'd suggest postponing this in favour of getting final task 1 test ready.

Yes that's okay. (I might hack something stupid together on Sunday to make sure my programs reacts to the tests as it should.)

[jGleitz]

I guess #122 should solve this completely.