Generate counterfactual for non ml model

[Himanshu-1988]

Hi, I am working on system in which i want to generate counterfactual ,as of now i have some rules that can map input to output which could be proxy to ml model predict function. I have designed my code using predictor wrapper function as suggested in document and able to train counterfactual proto model, but while generating counterfactual I am getting error.

#rule which can be proxy to ML model
def get_similarity(data_input):   
  #scaled_dangerours_arc_np = sc.transform(dangerous_arc_1.reshape(1, len(dangerous_arc_1)))
  scaled_data_input = sc.transform(data_input)
  scaled_mapping_data = sc.transform(mapping_data.to_numpy())

  cs = cosine_similarity(scaled_data_input,scaled_mapping_data) 

  if cs[0][0] > .99:
    print(cs[0][0])
    return np.array([0,1]).reshape(1,2)
  else :
    print(cs[0][0])
    return np.array([1,0]).reshape(1,2)

# Define prediction function as per library
predictor = lambda x: get_similarity(x)

# initialize and fit the explainer
cf = CounterfactualProto(predictor, shape, use_kdtree=False, theta=10., max_iterations=1000,
                         feature_range=(X_train.min(axis=0), X_train.max(axis=0)), 
                         c_init=1., c_steps=10)

cf.fit(X_train)

Out put :

Out[58]: CounterfactualProto(meta={
  'name': 'CounterfactualProto',
  'type': ['blackbox', 'tensorflow', 'keras'],
  'explanations': ['local'],
  'params': {
              'kappa': 0.0,
              'beta': 0.1,
              'gamma': 0.0,
              'theta': 10.0,
              'cat_vars': None,
              'ohe': False,
              'use_kdtree': False,
              'learning_rate_init': 0.01,
              'max_iterations': 1000,
              'c_init': 1.0,
              'c_steps': 10,
              'eps': (0.001, 0.001),
              'clip': (-1000.0, 1000.0),
              'update_num_grad': 1,
              'write_dir': None,
              'feature_range': (array([ 9.08163548e+00, -3.54359300e+02,  1.04347277e+01,  0.00000000e+00,
       -3.54359300e+02,  0.00000000e+00,  1.90000000e+01,  0.00000000e+00,
        1.09790468e+01,  6.89573002e+00,  0.00000000e+00,  0.00000000e+00,
       -1.81376961e-01,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
        1.22245255e+01,  4.75794268e+00,  0.00000000e+00,  0.00000000e+00,
        7.64921904e+00,  9.05236244e+00,  0.00000000e+00,  0.00000000e+00,
        8.61718941e+00]), array([4.75143848e+03, 2.67172000e+02, 4.62114893e+03, 1.24476924e+03,
       2.67172000e+02, 1.44829879e+03, 6.30000000e+01, 2.60000000e+02,
       3.16285059e+03, 4.48280762e+03, 1.44829879e+03, 6.00000000e+01,
       2.04271900e-01, 5.00000000e+01, 1.10000000e+01, 1.01792860e+01,
       4.63664893e+03, 4.96300977e+03, 2.00000000e+00, 1.29328054e+02,
       4.68819727e+03, 4.57909912e+03, 1.44829879e+03, 1.10000000e+01,
       4.62135449e+03])),
              'shape': (1, 25),
              'is_model': False,
              'is_ae': False,
              'is_enc': False,
              'enc_or_kdtree': False,
              'is_cat': False,
              'trustscore_kwargs': None,
              'd_type': 'abdm',
              'w': None,
              'disc_perc': (25, 50, 75),
              'standardize_cat_vars': False,
              'smooth': 1.0,
              'center': True,
              'update_feature_range': True}
            ,
  'version': '0.9.0'}
)

# generate a counterfactual
X_cf = X_test[2].reshape(1,len(X_test[2])) 
explanation = cf.explain(X_cf,verbose=True)

ERROR :  

ValueError                                Traceback (most recent call last)
<command-2270166967784462> in <module>
      1 # generate a counterfactual
      2 X_cf = X_train[2].reshape(1,len(X_train[2]))
----> 3 explanation = cf.explain(X_cf,verbose=True)
      4 
      5 print(f'Original prediction: {explanation.orig_class}')

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a0883ca0-0fac-44f2-9b48-e8a1f1ba45b2/lib/python3.8/site-packages/alibi/explainers/cfproto.py in explain(self, X, Y, target_class, k, k_type, threshold, verbose, print_every, log_every)
   1352         # find best counterfactual
   1353         self.best_attack = False
-> 1354         best_attack, grads = self.attack(X, Y=Y, target_class=target_class, k=k, k_type=k_type,
   1355                                          verbose=verbose, threshold=threshold,
   1356                                          print_every=print_every, log_every=log_every)

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a0883ca0-0fac-44f2-9b48-e8a1f1ba45b2/lib/python3.8/site-packages/alibi/explainers/cfproto.py in attack(self, X, Y, target_class, k, k_type, threshold, verbose, print_every, log_every)
   1126                         X_der_batch = np.concatenate(X_der_batch)
   1127                         X_der_batch_s = np.concatenate(X_der_batch_s)
-> 1128                         grads_num = self.get_gradients(X_der_batch, Y, cat_vars_ord=self.cat_vars_ord,
   1129                                                        grads_shape=pert_shape[1:]) * c
   1130                         grads_num_s = self.get_gradients(X_der_batch_s, Y, cat_vars_ord=self.cat_vars_ord,

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a0883ca0-0fac-44f2-9b48-e8a1f1ba45b2/lib/python3.8/site-packages/alibi/explainers/cfproto.py in get_gradients(self, X, Y, grads_shape, cat_vars_ord)
    911             grads[idx_nograd] = np.zeros(grads.shape[1:])
    912         grads = np.mean(grads, axis=0)  # B*F
--> 913         grads = np.reshape(grads, (self.batch_size,) + grads_shape)  # B*(shape of X[0])
    914         return grads
    915 

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a0883ca0-0fac-44f2-9b48-e8a1f1ba45b2/lib/python3.8/site-packages/numpy/core/overrides.py in reshape(*args, **kwargs)

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a0883ca0-0fac-44f2-9b48-e8a1f1ba45b2/lib/python3.8/site-packages/numpy/core/fromnumeric.py in reshape(a, newshape, order)
    296            [5, 6]])
    297     """
--> 298     return _wrapfunc(a, 'reshape', newshape, order=order)
    299 
    300 

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a0883ca0-0fac-44f2-9b48-e8a1f1ba45b2/lib/python3.8/site-packages/numpy/core/fromnumeric.py in _wrapfunc(obj, method, *args, **kwds)
     55 
     56     try:
---> 57         return bound(*args, **kwds)
     58     except TypeError:
     59         # A TypeError occurs if the object does have such a method in its

ValueError: cannot reshape array of size 0 into shape (1,25)

Can you please help me on this

[jklaise]

@Himanshu-1988 I've added syntax highlighting to your post to help debug easier :)

It would be good to know more information about the types, e.g. specifically the data_input of the get_similarity function and also further parameters such as shape. To work with the method it's expected that data_input be an np.ndarray with the leading dimension reserved for batch (i.e. the predictor should be able to operate on data with arbitrary batch size N just like any ML model).

That being said, gradient based counterfactual methods such as this are not well suited for non-gradient-based ML models/functions because they implicitly assume that the output of the model/function is differentiable wrt to the input which it isn't in your case (because there's custom logic with if/else), so I wouldn't expect this to give good results. Instead, I would recommend looking at CFRL which is specifically designed to work on non-differentiable models/functions.

[Himanshu-1988]

Thanks for your response.

shape : its a tuple of (1,25) data_input : numpy.ndarray with shape (1, 25)

From your response its clear to me that I can build a system that can generate counterfactual for non-ml model. I will start exploring CFRL

Thanks

[Himanshu-1988]

Thanks I am able to generate counterfactual for non ml model using CFRL