develop testing conventions - when do we want hidden vs visible tests

[lwasser]

RElated to the discussion yesterday, we should decide what types of tests are

1. open tests that the students can see vs
2. closed hidden tests that are run by the instructor

Visible Tests (open)

General structural tests that are asserts, true/false and generally there to guide the students along in the assignment.

• plot type
• plot contains x and y axis labels
• plot has a title
• data are of type -- (e.g. data frame, numpy array, etc)
• linting
• imports in header are all used in notebook

Hidden Tests

I think many tests will be hidden. these tests relate specifically to the assignment an associated data. They are not tests surrounding structural things like (the data are a data frame or the plot has image data in it).

• essentially anything specific to the homework like
• plot title contains NAIP
• plot data are correct (data need to be calculated for this)
• plot labels contain
• data submitted are correct

[kcranston]

We can also put all of the helper function definitions in hidden cells (run_tests and output_results). Eventually, these will move to a separate library, but for now this will reduce the complexity of the student notebook.

[lwasser]

oh... but if they are hidden, they won't run for the students right? so that means no visible tests (which is ok to get started!)

i also thought about simple test suites for the basic stuff like a plot has a title and x y labels. i have a lot of ideas after yesterday- it was so helpful @kcranston i'm glad you pushed for it.