Improve feedback on exercises

[edoardob90]

Currently, we're using pytest to test users' answers against reference solutions. However, we want to enhance the feedback provided to users when their answer is incorrect or doesn't match the reference solution. We would like to explore alternative approaches that encourage users to explore diverse solutions and improve their coding skills while still receiving helpful guidance.

Here are some potential approaches to consider:

1. Compare function signatures (we're sort of already doing that)
2. Leverage Abstract Syntax Trees (AST)
3. Test multiple reference solutions
4. Create custom pytest assertions

A couple of comments to (2) and (4).

Abstract Syntax Trees (AST)

We can use Python's ast module to parse users' code and the reference solutions into ASTs. By comparing the structural similarities between the two ASTs, we can provide a useful feedback while allowing for diverse solutions. This encourages users to follow good coding practices without adhering to a specific solution.

Custom pytest assertions

Develop custom pytest assertions that evaluate users' code based on specific criteria, such as the proper use of loops, conditional statements, and other coding constructs. This way, we can provide feedback based on the user's understanding and application of core programming concepts.

[baffelli]

I agree with the AST solution as mentioned today. Still it's important to understand how to diff trees meaningfully but could be a gun challenge

[despadam]

After the refactoring of the testsuite (see https://github.com/empa-scientific-it/python-tutorial/commit/4dc3a7e20936fc5f62e806d7a4818d1289838698), we now sometimes get cryptic error messages. Participants have mentioned that it is not very clear to them what the error is. Perhaps we need to bring back some of the previous funtionality - written in a better way of course and respecting the code's structure after the refactoring.

[despadam]

I am not sure whether captured output and captured error work as expected.

In the example below, 'a' is printed outside of the captured output box and 'b' is not printed at all.

[baffelli]

I know this is confusing, but it is due to the complex lifecycle of ipytest. Let's distinguish two cases:

1. Everything outside of the function definition gets executed in the main Ipython kernel and therefore is printed immediately in the Ipython cell output.
2. Everything inside of the function definition solution_greet only gets printed when the function gets run through pytest in a separate process. However, I think you are right and the output doesn't get captured properly at the moment