datacamp / pythonwhat

Verify Python code submissions and auto-generate meaningful feedback messages.
http://pythonwhat.readthedocs.io/
GNU Affero General Public License v3.0
69 stars 31 forks source link

Document the order to check chained functions #370

Open richierocks opened 5 years ago

richierocks commented 5 years ago

In ch3ex6 of "Machine Learning for Time Series in Python", the student has to complete this line of code.

missing_values = prices.isna().sum()

The SCT to check this was

Ex().multi(
    check_correct(
        check_object("missing_values").has_equal_value(), 
        multi(
            check_function("prices.isna.sum", signature=False), 
            check_function("prices.isna")
        )
    )
)

This gives a bad feedback message when the student types a different function to is.na().

For example, the submission

missing_values = prices.isnaxxx().sum()

Results in the feedback

Did you call `prices.isna.sum()`?

This is confusing: it ought to say

Did you call `prices.isna().sum()`?

A partial fix is to perform the checks in reverse order. That is functions should be checked from left to right, not right to left.

Ex().multi(
    check_correct(
        check_object("missing_values").has_equal_value(), 
        multi(
            check_function("prices.isna"),
            check_function("prices.isna.sum", signature=False)
        )
    )
)

In this case, the submission

missing_values = prices.isnaxxx().sum()

gives the feedback

Did you call `prices.isna()`?

It isn't perfect though. The submission

missing_values = prices.isna().sumxxx()

gives the feedback

Did you call `prices.isna.sum()`?

So a custom missing message needs to be provided.

Ex().multi(
    check_correct(
        check_object("missing_values").has_equal_value(), 
        multi(
            check_function("prices.isna"),
            check_function("prices.isna.sum", signature=False, missing_msg = "Did you call `prices.isna().sum()`?")
        )
    )
)

Since this is a subtle problem, I think it needs documenting. Ideally, the feedback message also needs fixing so that parentheses are included where the intermediate element is a function.