Roundtrip tests with missing values

[jorisvandenbossche]

Currently the roundtrip tests (https://github.com/data-apis/dataframe-interchange-tests/blob/8c1fc08716629ad4bcbb089b2a961da13c0e234c/tests/test_from_dataframe.py#L14) only use a simple data generation logic, which doesn't include any missing values, as far as I see:

https://github.com/data-apis/dataframe-interchange-tests/blob/8c1fc08716629ad4bcbb089b2a961da13c0e234c/tests/strategies.py#L75-L87

[honno]

Yeah this would be neat. Quite possible but a bit tricky to implement, given we'd need to figure out what protocol adopters allow missing values in which scenarios (i.e. dtypes) for an initial "missing values mask" (not necessarily an actual mask, just what we use on our end to figure out where missing values exist for a given test example), and a translation of said mask for every protocol adopter in wrappers.py.

[jorisvandenbossche]

Another option could be to not create numpy arrays (like in the snippet above), but plain python lists of scalar values for a column, and then let each library convert that to their native representation in wrappers.py mock_to_toplevel. For example, then we could say that missing values are represented as None (eg ["a", "b", None] for strings). That avoids needing a sentinel or mask approach to make it work with numpy arrays.

[honno]

That avoids needing a sentinel or mask approach to make it work with numpy arrays.

Ah I wasn't concerned about masking the original numpy arrays, as I assume it's easier to create an initial test example (an adopter's native column object) and then create/manipulate a new test example from that with "randomly" placed missing values (either/both sentinel or mask where appropriate).

Another option could be to not create numpy arrays (like in the snippet above), but plain python lists of scalar values for a column,

Just to note I initially was going to generate plain Python lists but found all the adopters consumed numpy arrays, so thought I could utilise Hypothesis' nps.arrays() tool to very easily generated complex test examples for all the dtypes.