Multiple feedback messages when submission has syntax error

[machow]

The first comes from the backend, and is passed to test_exercise. The second is from running the SCT.

Should...

1. add test_error function, which fails if backend sent in an error.
2. ensure that if SCT completes, and there is already an error, then it fails and returns the original error message.

Note that this does not include the case where someone wants an exercise to succeed, even if there is a syntax error (we can always through in an allow_error function if it comes to it, seems unlikely).

[filipsch]

@machow I've made a proposal for this issue in the google doc I've shared with you (not sharing here because this is a public repo); I think this will be the cleanest solution of all other solutions, that requires the least amount of extra work on the campus-app side, and keeps it consistent for the frontend side with other languages.

[filipsch]

Coordinate with cc @rv2e on this

[machow]

@filipsch, okay I've read the proposal and think it makes sense. If I understand it correctly, then the original steps conform to the proposal, yeah? It sounds like I should also make sure the original backend-error stays in the output, so [backend-error, sct-fail] is an acceptable payload. (the frontend would just show the sct message.

[filipsch]

@machow you're confusing error and backend-error I think. backend-error is a real failure of the backend, and is a special case (because the backend should never fail, hopefully).

The first usecase (of runSubmit() in Exercise.py is):

• runRun() produces a query result (user types valid SQL code that runs without error). The payload up to now is [ query_result ]
• Next, sqlwhat is called, and it seems that user was right. It returns a [ sct ] payload which contains correct: true and a congratulatory message.
• The two payloads are appended in sqlbackend, which leads to [ query_result, sct ].
• If sent to the frontend, there's no issue; both can be shown: the query result, and correct exercise sidebar overlay.

The second usecase (of runSubmit() in Exercise.py is):

• runRun() produces a query result (user types valid SQL code that runs without error). The payload up to now is [ query_result ]
• Next, sqlwhat is called, and it seems that user made mistake. It returns a [ sct ] payload which contains correct: false. The message of the SCT contains more info about the mistake the user made.
• The two payloads are appended in sqlbackend, which leads to [ query_result, sct ].
• If sent to the frontend, there's no issue; both can be shown: the query result, and an incorrect submission feedback block.

The third usecase of runSumbit() in Exercise.py is:

• runRun() produces an error because a user made a mistake; the payload is now [ error ].
• Next, sqlwhat is called, and it sees that the user made an error. It returns a [ sct ] payload which contains correct: false. The message of the sct contains the error the user made (just copied over from the error payload), with potentially additional information to fix it (depends on how smart sqlwhat is).
• The two payloads are appended, and this leads to [ error, sct ] in the sqlbackend
• Because this would lead to two outputs in the 'incorrect submission feedback block' on the frontend, I would say that the backend drops the error payload, and only sends back [ sct ]. The SCT message will contain the error made anyways.

The fourth usecase of runSubmit() in Exercise.py is:

• any of the steps produce an error in the sqlbackend or the sqlwhat parser that is not handled gracefully, which leads to a backend-error.
• The payload sent to the frontend will be [ backend-error ], and, for now, the frontend simply displays it as an incorrect message, so you have access to the error, but at some point we should message this differently. That's up to the frontend to do.

I would really do it like this. If it is still not clear, don't build it yet. cc @machow @rv2e

[machow]

in production w/ implementation of test_error (not that we changed the behavior to communicating error and sct type messages in different places, so did not need to select only one message to display).