Ignore NaN values in validation

[diegoquintanav]

This is related to https://github.com/TMiguelT/PandasSchema/issues/5

Basically, Whenever I add some validation like

import pandas schema as ps
# mumble mumble
ps.Column('some_column', [ps.validation.InListValidation(["FOO", "BAR", "BAZ", "nan"])]),

I have to add the "nan" to the list to avoid having null rows being assumed non valid, considering that some rows might contain null values or None. I understand that one might want to mark null values as invalid, so I propose different interface, something like

ps.Column('some_column', [ps.validation.InListValidation(["FOO", "BAR", "BAZ"])], ignore_nan=True),

This parameter ignore_nan can be set as False by default to get the current behaviour.

[RoyalTS]

I like this idea. Seems fundamentally wrong to be testing against nan after coercion. Would you consider a pull request along these lines?

[multimeric]

Oh, yes this seems pretty reasonable. I'd definitely accept a PR for this

[multimeric]

Wait, I just realised that this seems like the use-case for the allow_empty argument in the column constructor.

If you do this, does it have your desired behaviour?

ps.Column('some_column', [ps.validation.InListValidation(["FOO", "BAR", "BAZ"])], allow_empty=True)

[RoyalTS]

Maybe. But if I set allow_empty=True I get TypeError: data type not understood.

[multimeric]

Okay, I'll have a look into why that's happening

[contang0]

perhaps rather than having an option to ignore nans, we should have an option to ignore a list of values? this is desirable in situations where you have data where nans are represented by blanks, spaces etc, and for whatever reason you do not want to convert them into proper NaNs. in that case i'd prefer to let the validator know which values i want to ignore. something like

ps.Column('some_column', [ps.validation.InListValidation(["FOO", "BAR", "BAZ"])], ignore_values=[np.nan, '', ' '])

[multimeric]

Possibly, although this could equally be done by composing Validators, and I prefer to keep the API simple if there's already a reasonable solution. In your case you could just have ps.validation.InListValidation(["FOO", "BAR", "BAZ", None, '', ' ']), and in a more complicated case like ps.validation.InRangeValidation(0, 100) you could use ps.validation.InRangeValidation(0, 100) | ps.validation.InListValidation([None, '', ' ']). See if something along those lines works, and if that still isn't satisfactory then let me know.