Adding support for testing stderr output

[RadeKornjaca]

Short description

Student assignments include error handling situations. These need to be prevented and program should terminate with status different to zero. Check if output on stderr is equal to the assignment text specified in those cases.

Full Description and Motivation

When using traditional *nix tool such as ls, the program prints certain message when trying to list a file/directory that doesn't exist. For example: ls no_such_file

Would print: ls: cannot access 'no_such_file': No such file or directory

Students are currently obliged to exit with certain error code in certain situations, and that aligns with the philosophy of the nix CLI tools that is the goal for them to learn. On the other hand, all standard nix tools incorporate the behavior to print some kind of message in case of error. The idea is to pursue that kind of feedback in student assignments. Take these examples, derived from the safe_fopen function:

if(fp == NULL) {
    printf("Hi, mom!\n");
    exit(error_code);
}

It could be written as this:

if(fp == NULL) {
    fprintf(stderr, "Can't open '%s'!\n", filename);
    exit(error_code);
}

The feedback message won't be of any help to the smoke_test itself, but it should help student when testing his/her solution, as it is helpful when using the CLI tools. I don't know about you, but I'm not thrilled always looking at the value stored in $? just to see if my last command was good. :) From the teaching perspective, I see it as a good way to discipline the student's coding skills, to see that every typed line has its purpose. Currently, the tool would forbid any content passed on stdout and stderr after it checks exit status, as far as I investigated. To give the student right mindset about how feedback should be done in *nix world, proposed solution is to test the output on stderr beside exit code status. It will also prevent the error text to be written in output file in case of redirecting the stdout using > or similar Bash operators. While I understand that these kind of tests could fail because of simple typing errors, the assistant should bear that in mind while grading the student and lower the score in those situations as little as possible.

Proposed solution

1. Add new fixture data type expected-stderr.
2. Read its value in tests/functional/file.py method _setup and put it to the dedicated attribute inside BaseFileTest class
3. In BaseFileTest class, override the _check_program_stderr to compare the normalized stderr and fixture data in the same manner as files are already compared in the current version
4. If successful, the _check_program_stderr will return True to _sanity_check, which will combine the results given from testing return code, stderr and stdout

Please note that I have used previous experience from #1 for assembling this solution proposal. It is only an idea that maybe would work.

[petarmaric]

Given your (excellent) lengthy proposal it may take me a bit more time to review it and document my observations, questions and recommendations.

However, while you wait can you please take a look at these alternatives to the

fprintf(stderr, char *format, ...);
exit(error_code);

mantra:

1. https://linux.die.net/man/3/errx
2. https://linux.die.net/man/3/error

They're fairly similar in the sense they both display an error on stderr and then exit the program.

While reviewing your proposal I've been analyzing the source code of core *nix utilities (such as ls, cat, and mkdir) , as implemented in FreeBSD and GNU core utils. From what I saw these functions are more commonly used for fatal-failure scenarios over the fprintf-exit mantra. However, they both seem to be non-standard extensions - be it BSD or GNU specific.

Would you mind diving deeper into what's considered a canonical (and cross-platform) way to properly "display-error-and-exit" in C?

[petarmaric]

Oh, and I forgot to mention this excerpt from error(3) man page:

error() is a general error-reporting function. It flushes stdout, and then outputs to stderr the program...

Given that stdout and stderr and usually multiplexed to the same terminal I fear that the fprintf-exit mantra may sometimes lead to their streams mixing on output.

[RadeKornjaca]

Regarding your questions, I have investigated and came with answers:

• Cannonical way of dealing with output is indeed using the fprintf to stderr and exit mantra
• In case of system errors such as a file opening error, the errno value from errno.h and strerror from string.h can be used in combination to describe the error
• stderr is unbuffered by default, while stdout is; that means all the error message has precedence if stdout is not previously flushed

Chapter 7.6 of Richie & Kernighan's Programming Language C book uses exactly the fprintf to stderr and exit to implement error handling in their cat program example. To complement the system errors, I suggest using errno for obtaining the error message for what really went wrong in program. Consider this example:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <errno.h>

extern int errno;

int main(char *argv[], int argc) {
    char filename[] = "no-such-file.txt";
    FILE *fp = fopen(filename, "r");

    if(fp == NULL) { 
        char *error_message = strerror(errno);
        fprintf(stderr, "Error opening file %s: %s\n", filename, error_message);
        exit(EXIT_FAILURE);
    }   

    fclose(fp);

    return EXIT_SUCCESS;
}

The already standard safe_fopen function could be modified to behave like this code. Expected output on stderr is:

Error opening file no-such-file.txt: No such file or directory

Regarding other types of non-system errors such as insufficient/too many arguments or divide by zero, fprintf with custom message printed to stderr and exit can be used.