Color Code Grading Script Output

[asayler]

There have been some requests to try to color code grading script output to make it more easily readable/parsable. How best to do this is an open question. Currently grading script output is interrupted as raw unformatted plain text. To add color or other formatting, we will likely need to agree upon a specific formatting language that grading scripts will output instead and that can be interpreted by the frontend for color coding purposes. Any formatting should also be human readable without the benefit of parsing or color coding to preserve compatible with non web-based frontends. Markdown is a good example of a language meeting this requirement.

If we want highliting the diffs between the student output and the expected output, we might want to look at using somthing like https://code.google.com/p/google-diff-match-patch/. Or we could just output in standard diff format and use an existing syntax highlighter like http://alexgorbatchev.com/SyntaxHighlighter/manual/brushes/diff.html (although standard diff format is line orinted, not charecter oriented, so it may be of limited use in these cases).

Comments or suggestions welcome.

[hexacyanide]

I noticed that this issue has been open for quite a long time now, and thought it might be a good idea to start further discussion on it. Although my knowledge of current practical use of COG is limited due to having only used it in testing (i.e. never in a real-world environment such as a computing course), it still seems that using some parts of it can be fairly inconvenient depending on how output scripts are written.

If this issue is still relevant, I'd like to propose several possible practical implementations for this feature. The main factors I have taken into consideration are convenience, usability, backwards compatibility, and extensibility. Since the grader's main target audience appears to be students, I have put greater weight on the first two factors.

---

I am assuming the features in consideration are the following:

• an integrated difftool (by line or character)
• syntax highlighting or coloring of output

Those pose the following questions or problems:

• how should the difftool be implemented?
  • should diffs be done by line or by character?
  • will they be computed on the server or the web interface?
• how can parsers robustly detect the type of output and handle it accordingly?
  • what formats should be supported?
  • how will colorizing output interact with diff output?
  • how should the parser detect legacy runs/submissions or legacy grading scripts?

Taking the above into consideration, I believe the most practical approach to an implementation would be some variant of one or multiple of the following:

• implementing a difftool into the web interface
  • it is used manually by the user, (no compatibility problems)
  • type of diff can be also specified by user (simple toggle would suffice)
  • can expand out below run output for simple text transfer
  • is the easiest to implement
  • does not consider colorized output

Or a slightly more complex solution:

• specifying additional metadata for tests and implementing client code to handle it accordingly:
  • requires the user to do less
  • is backwards-compatible, could work well with Markdown
  • can be extended outside of just simple markdown

The two options aren't mutually exclusive, so they could potentially work well together. The second option also probably requires a bit more consideration on some details of an implementation, such as:

• how should a client detect a custom output type?
• should the type be specified during assignment creation or should it be written in the grading script?

Overall, I think it'd be best to compute diffs/parse output on the user-side (browser). A simple output_fmt property for assignments or a simple output header for an assignment could be easily implemented, and client handling (if written properly) would be backwards-compatible. I also believe Markdown would be a good choice for these implementations.

Looking forward to hearing more thoughts on this matter.

[asayler]

I like the idea of making the COG web interface (optionally) understand Markdown, but that will require the inclusion of the client-side (e.g. JS) markdown processor. I definitely don't want to process markdown on the server, since I like the COG server being largely agnostic to the format of the output (necessary to ensure we can support alternate frontends like the COG CLI). I'm happy to add a "format" metadata field to the server, but beyond that, I'd prefer not to have the server involved with processing grading script output directly.

To make the web interface process diffs, we'd probably need to extend markdown syntax to have some kind of diff field. e.g. maybe using four-backticks instead of the normal three-tick source code operator, i.e.

<four-backticks>
<output A>
<four-backticks>
<output B>
<four-backticks>

But that now means both using a client-side markdown processor and modifying it to speak our extended version of markdown, which may be getting a bit complex. I'm also not sure how much this will get used, since it increases the complexity of the grading scripts themselves which now must output Markdown instead of just plaintext. So we may want to sit on this a bit more, or come up with a simpler rev 0.1 approach to prove out the demand.