Unexpected parameter value passed into Test

[elutins]

I am trying to write a TestSuite, in this example I will reference the TestShareOfOutListValues test in particular, but question applies to all tests. I am trying to specify the cat_stattest and cat_stattest_threshold parameters for the test, but get a TypeError: __init__() got an unexpected keyword argument 'cat_stattest'.

This page in the documentation refers to the TestShareOfOutListValues accepting a cat_stattest and cat_stattest_threshold parameter

here is my code:

test_suite = TestSuite(tests=[
    TestShareOfOutListValues(column_name=column, cat_stattest="jensenshannon", "cat_stattest_threshold"=1)
])
test_suite.run(reference_data=test_ref, current_data=test_current, column_mapping=ColumnMapping(categorical_features=[col])

[elenasamuylova]

Hi @elutins,

The TestShareOfOutListValues does not accept this parameter.

I double-checked the docs (the Test is listed under the Data Quality group). It appears to be reflected correctly:

For this specific Test, you can pass:

• the standard test condition parameters (such as "less than" (lt), "greater than" (gt) and so on - documented here).
• a test-specific parameter where you pass a list of values - this is relevant if you do not pass the reference dataset and want to manually specify the list of acceptable values.

I noticed you linked to the "Presets" subheading on the page. In this case, the listed parameters refer to the Test Preset. If you pass the parameters to the Preset, they apply only to the relevant individual Tests inside it.

For example, here is the description of the NoTargetPerformanceTestPreset

It includes the TestShareOfOutListValues we discuss, but also includes the TestShareOfDriftedColumns that detects distribution drift. The cat_stattest parameter is specific to it. If you pass the cat_stattest parameters to the Preset, it will affect the drift detection method for categorical features. For this parameter, you must pass one of the applicable values (documented here, such as psi, jensenshannon, etc.) It will not affect other Tests in the Preset - only the ones that refer to data drift detection.

Generally, since you design custom Test Suites from individual Tests, you can safely ignore everything that relates to the Presets and Preset parameters.

Could you share what is the expected behavior you'd like to see when passing the parameter to the TestShareOfOutListValues test?

The default behavior is to fail if at least one value is out of the list. If you want to set a different condition, you can use the standard parameters like (lt, gt, lte, gte and so on).

[elutins]

hi @elenasamuylova

thank you for clarifying this was super helpful; I believe the confusion arose from the difference between the TestPresets and an individual Test.

Follow up question, for test such as TestShareOfOutListValues, I am noticing that TestShareOfOutListValues is failing because the current dataset contains missing values whereas the reference dataset does not. Is there a way to exclude missing values from being counted in TestShareOfOutListValues test? I already have a TestColumnShareOfMissingValues to handle checks for missing values.

[elenasamuylova]

Hi @elutins,

I am afraid the test explicitly checks for all new values, and this includes "unknowns" or "nulls". Possible workarounds are:

• Make sure the reference dataset includes missing values
• Pass a custom list of allowed values, including missing values