search

istopwg / ippeveselfcert

IPP Everywhere Printer Self-Certification Tools
https://www.pwg.org/ipp/everywhere.html
Apache License 2.0
21 stars 6 forks source link

Update test scripts to log all output to a file in addition to writing to stdout / stderr #65

Closed wifiprintguy closed 4 years ago

wifiprintguy commented 4 years ago

Frequently, those running the tests will just run them, and if there are failures, will provide the plist files, which are not very easily read by humans. The person running the test could run the test script like so on UNIX-like operating systems, but they would have to be somewhat savvy and/or the instructions would need to be updated, for the output of the scripts to be captured to a log file that could also be shared back to the PWG or the company's IPP leads:

$ ./ipp-tests.sh "My Cool Printer" | tee ipp-tests-$(date +"%Y%m%d%H%M").log 2>&1

It would be better if the scripts wrote their trace messages to a log file so that those running the tests would not need to go to great efforts to learn how to capture the output, but would instead just run the scripts and have the logs available if needed.

Update the scripts to capture the output. If that is not practical, then provide instructions for how to run the scripts.

michaelrsweet commented 4 years ago

@wifiprintguy Unfortunately, I don't think this is feasible on Windows for sure. The DNS-SD script would also likely require significant changes on Linux/macOS...

Perhaps an alternative is to provide a program (or an option to ippevesubmit) that effectively "replays" the tests from the plist file, showing the output in the plaintext format that ipptool produces?

wifiprintguy commented 4 years ago

@wifiprintguy Unfortunately, I don't think this is feasible on Windows for sure. The DNS-SD script would also likely require significant changes on Linux/macOS...

I'm experimenting with updating dnssd-tests.sh / ipp-tests.sh / document-tests.sh using the methodology in this stackoverflow thread:

https://stackoverflow.com/questions/314675/how-to-redirect-output-of-an-entire-shell-script-within-the-script-itself

If it works out I'll let you know how it goes. The Windows users may just have to live without it unless they install the Windows Subsystem for Linux (https://docs.microsoft.com/en-us/windows/wsl/install-win10). Moving to that for Windows support might also mean we can simplify our instructions and not maintain two sets of scripts.

Perhaps an alternative is to provide a program (or an option to ippevesubmit) that effectively "replays" the tests from the plist file, showing the output in the plaintext format that ipptool produces?

Maybe if my experiments don't pan out we can pursue that. That sounds like it is more work than just updating the shell scripts but I could be wrong.

michaelrsweet commented 4 years ago

@wifiprintguy That stack overflow article appears to be focused on some bash'isms for redirection, which don't work for /bin/sh (POSIX shell).

As for doing a replay option/command, we already have 90% of the code needed in ippevesubmit so it would just be a matter of writing the test name with a pass/fail followed by any errors. It should literally take me 30 minutes or less.

michaelrsweet commented 4 years ago

[master dc6b5b6] Add '-r' (replay) option to ippevesubmit (Issue #65)

@wifiprintguy Let me know if you'd like to see any other functionality here... I'm tracking down a REGEX bug that showed up when I started testing the upcoming build...

wifiprintguy commented 4 years ago

@michaelrsweet, can you provide an example of how the "-r" option works? Do I need to have all the plist files and the models.txt to use it?

wifiprintguy commented 4 years ago

Ah, never mind, I rebuilt and tried it out. It works well!