Deal with missing values

[londumas]

There does not seem to be a support for missing values currently. For example dealing with the 'Age' feature in the Titanic dataset:

import pandas as pd
import sklearn
import catboost
import dice_ml

def full_data_set(dataset='train'):
    df = pd.read_csv(
        "../{}.csv".format(dataset))
    if dataset=='test':
        df_truth = pd.read_csv(
        "../gender_submission.csv")
        df = pd.concat([df, df_truth],axis=1)

    ###
    for c in ['Cabin','Embarked']:
        df[c] = df[c].apply(str)

    drops = ['PassengerId','Name','Ticket','Cabin','Embarked']
    df = df.drop(drops,axis=1)

    return df

dataset = full_data_set('train')

###
X_train = dataset.copy()
X_train = X_train.drop(['Survived'],axis=1)
y_train = dataset['Survived']

cat_features = ['Sex']
cat_features_index = [i for i, col in enumerate(X_train.columns) if col in cat_features]

model = catboost.CatBoostClassifier(iterations=200,
                           depth=5,
                           learning_rate=1,
                           loss_function='Logloss',
                           verbose=20,
                           task_type="CPU",
                           devices='0',
                           cat_features=cat_features_index)
# train the model
_ = model.fit(X_train, y_train)

###
dataset = full_data_set('train')

continuous_features = [ el for el in dataset.columns if (el!='Survived') and (el not in cat_features)]
print("continuous_features = ",continuous_features)

d = dice_ml.Data(dataframe=dataset,
                 continuous_features=continuous_features,
                 outcome_name='Survived')

# provide the trained ML model to DiCE's model object
backend = 'sklearn'
m = dice_ml.Model(model=model, backend=backend)

# initiate DiCE
exp_random = dice_ml.Dice(d, m, method="random")

for idx in [0, 888]:

    query_instances = X_train[idx:idx+1]

    # generate counterfactuals
    dice_exp_random = exp_random.generate_counterfactuals(query_instances,
        total_CFs=4, desired_class="opposite", verbose=False)

    dice_exp_random.visualize_as_dataframe(show_only_changes=True)

0:  learn: 0.4365535    total: 54.3ms   remaining: 10.8s
20: learn: 0.2308290    total: 100ms    remaining: 855ms
40: learn: 0.1532110    total: 135ms    remaining: 524ms
60: learn: 0.1088515    total: 166ms    remaining: 379ms
80: learn: 0.0825711    total: 195ms    remaining: 286ms
100:    learn: 0.0716318    total: 221ms    remaining: 217ms
120:    learn: 0.0629166    total: 247ms    remaining: 162ms
140:    learn: 0.0562215    total: 274ms    remaining: 115ms
160:    learn: 0.0522724    total: 302ms    remaining: 73.1ms
180:    learn: 0.0480928    total: 329ms    remaining: 34.6ms
199:    learn: 0.0464085    total: 355ms    remaining: 0us
continuous_features =  ['Pclass', 'Age', 'SibSp', 'Parch', 'Fare']
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.12it/s]
Query instance (original outcome : 0)
   Pclass   Sex   Age  SibSp  Parch  Fare  Survived
0       3  male  22.0      1      0  7.25         0

Diverse Counterfactual set (new outcome: 1.0)
  Pclass     Sex   Age SibSp Parch    Fare Survived
0      -  female     -     -     -  112.29        1
1      -  female     -     -   5.0       7        1
2      -       -  15.5     -     -   449.6        1
3      -  female     -     -     -  431.99        1
  0%|                                                                                                                                                                                                                                                               | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "conterfactual-malwares-titanic.py", line 67, in <module>
    total_CFs=4, desired_class="opposite", verbose=False)
  File "<home>/.local/lib/python3.7/site-packages/dice_ml/explainer_interfaces/explainer_base.py", line 102, in generate_counterfactuals
    **kwargs)
  File "<home>/.local/lib/python3.7/site-packages/dice_ml/explainer_interfaces/dice_random.py", line 171, in _generate_counterfactuals
    final_cfs_df_sparse, test_instance_df, posthoc_sparsity_param, posthoc_sparsity_algorithm)
  File "<home>/.local/lib/python3.7/site-packages/dice_ml/explainer_interfaces/explainer_base.py", line 420, in do_posthoc_sparsity_enhancement
    diff = query_instance[feature].iat[0] - int(final_cfs_sparse.at[cf_ix, feature])
ValueError: cannot convert float NaN to integer

[gaugup]

@londumas does some sample of the query_instance has a missing value? I think if dice-ml doesn't handle this scenario it is ok. Shouldn't the user woryy about supplying a legitimate value for all columns in query_instance? How would dice-ml go about generating CFs if the value in a column is not known apriori? We should probably raise an exception that the query_instance has missing values rather than erroring out during generation of counterfactuals.

Regards,

[candalfigomoro]

@gaugup I think a missing value is a legitimate value. Packages such as xgboost and lightgbm support features with missing values. I think DiCE should also handle missing values without having to impute them.

[urigott]

I came across this issue as well. In many cases missing values are very informative (just as much as "real" values), and since DiCE can handle models that accept missing values (such as LGBM, XGboost, CatBoost), it would be great if it was capable of handling missing values.