Where the status is error (the tests fail to execute cleanly), the top level message key should be provided. It should provide the occurring error to the user. As it is the only piece of information a user will receive on how to debug their issue, it must be as clear as possible. For example, in Ruby, in the case of a syntax error, we provide the error and stack trace. In compiled languages, the compilation error should be provided. The top level message value is not limited in length.
and
The per-test message key is used to return the results of a failed test. It should be as human-readable as possible. Whatever is written here will be displayed to the student when their test fails. If there is no error message, either set the value to null or omit the key entirely. It is also permissible to output test suite output here. The message value is not limited in length.
The test runner should strip some path information from failure messages and stack traces.
From interface.md:
and