test_function_v2

[hugobowne]

Hi!

I have been playing around with test_function_v2() and I like it.

• I have NOT been using it very much because I think there was a misunderstanding: I thought that I was to use it ONLY when both keyword/positional solutions were to be accepted; I didn't realize that it would push out test_function eventually

There are a couple of issues I have that we should discuss before I can really use it. i'm not entirely comfortable with the feedback messages that it provides:

E.g. it seems to always mention the keyword in question. See this:

course: https://campus.datacamp.com/courses/1533/3850?ex=5 repo: https://github.com/datacamp/HBA-python-problem-playground/blob/master/chapter1.md

The student may not have known the keyword file even existed. Could it provide i) a similar message as test_function in the case that the student uses a positional argument and ii) a different message if they use the keyword?

Also, will it be possible for an SCT to accept ONLY positional or ONLY keyword args? Apologies if this exists and I missed it

Another issue is that the SCT writer needs to write out explicitly ALL the arguments to be tested. It would be easier if it were like test_function(), in that it tests all those used in the solution code.

I am still coming to terms with test_function_v2. There are many subtleties in figuring out how to instruct students in this type of pedagogy.

Until now, I have been relying on my interns primarily for SCTs and have now realized that I need to get down'n'dirty w/ pythonwhat. Looking forward to it but wanted to let you know that other things may crop up

[filipsch]

@hugobowne

• About the messaging: I can add an argument that specifies for each argument whether the keyword should be explicitly mentioned. It would default to mentioning the keyword argument if it is possible to specify the argument by its keyword, and to not mentioning it if you can only specify the argument positionally (this information follows from the signatures of the function calls). We can also say it defaults to not mentioning the keyword argument at all, unless you tell it to. What do you think?
• About only accepting positional or only keyword args: I'm not a fan of that. If python allows you to specify the argument by keyword or by position, it should be accepted. If it's valid python code, pythonwhat should accept the call, and not force people into doing one or the other. What do you think?
• About explicitly writing out all arguments to be tested: I thought the entire disadvantage about test_function() was that you didn't have to be explicit about arguments in the first place. This made it hard to check what you were actually testing (arguments? keywords? which ones?) and also made it hard to specify argument/keyword-specific incorrect messages. Now, in test_function_v2(), it's clean. You specify a list of parameter names, and you can specify a list of incorrect messages with the same length, a list of do_eval values with the same length, etc. That way, there's no confusion, you have to be specific about what you're testing but I think that's a good thing. What do you think?

[filipsch]

@hugobowne Can you give your thoughts on this thing? The longer I wait with improving test_function_v2(), the longer there will be two "test_function code bases" that I have to maintain and support. And the longer it will take to switch from one system to the other. I'd like to have a clear, unambiguous interface for testing function calls at some point.

[hugobowne]

@filipsch I appreciate your patience with this. This week I am spending a lot of time with pythonwhat and will try my hand at working with both test_function() and test_function_v2() a great deal. I'll let you know how I go by EOW. That cool?

[hugobowne]

@filipsch i'll get onto this when the new backend is rolled out

[filipsch]

@hugobowne I suggest I provide another argument, named, that you can pass a list of boolean values, with the same length as the parameters you're specifying. If it's specified, it overrides the default behavior of mentioning the parameter name if it is possible to mention it by it's name.

Say, for example, you're testing

import pandas as pd
pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})

But the student did:

import pandas as pd
pd.DataFrame({'a': [10, 11, 12], 'b': [13, 14, 15]})

Depending on the SCT, you get different feedback:

test_function_v2("pandas.DataFrame", params = ["data"])
# message: Did you call `pd.DataFrame()` with the correct arguments? The argument you specified for `data` seems to be incorrect.

test_function_v2("pandas.DataFrame", params = ["data"], named = [True])
# message: Did you call `pd.DataFrame()` with the correct arguments? The argument you specified for `data` seems to be incorrect.

test_function_v2("pandas.DataFrame", params = ["data"], named = [False])
# message: Did you call `pd.DataFrame()` with the correct arguments? The first argument seems incorrect.

[hugobowne]

@filipsch my initial reaction is that I like this. Let me know about it. Due to delays, I will be getting to all of this next week and have my thoughts together by the DC retreat: is that ok?

[filipsch]

So I'll wait with implementing this, right?

[hugobowne]

I don't have enough experience yet with test_function_v2() to know what is a good idea so I think so, yes, wait.

[machow]

Closing, as this can be done in pythonwhat v2 using,

Ex().check_function('pandas.DataFrame', 0, signature=False).check_args(0).has_equal_value()

which will get a message like, "Check the first argument of the first function call for pandas.DataFrame".