search

DataCanvasIO / DeepTables

DeepTables: Deep-learning Toolkit for Tabular data
https://deeptables.readthedocs.io
Apache License 2.0
672 stars 118 forks source link

Integrated Gradients and Embedding layer #37

Open gladomat opened 4 years ago

gladomat commented 4 years ago

I would like to calculate feature importances using integrated gradients, as I am hoping this calculation will be faster than the SHAP KernelExplainer. Unfortunately, the embeddding layer is non-differentiable, since it's a simple matrix multiplication. This causes a failure in the gradient calculation.

I also tried embedding the categorical variables separately and then feeding the full data into the model. Theoretically, this procedure avoids calculating the gradients for the embedding layer because there is none present. However, when every variable is numeric the preprocesser discards all variables which are 0 from top to bottom. This is not what I want, as I want to train in batches.

I have two questions:

  1. Is there a way to calculate integrated gradients with embedding layers?
  2. How can I stop the deeptables pre-processor to discard variables which have non-unique (all zeros) values?

Thanks!

jackguagua commented 4 years ago

  • Is there a way to calculate integrated gradients with embedding layers? Very promising idea, looking forward to your progress.

  • How can I stop the deeptables pre-processor to discard variables which have non-unique (all zeros) values? Now you can try to set auto_discard_unique=False in ModelConfig which should solve the problem.