error in permutation feature importance result (ablation method)

timeseriesAI / tsai

Time series Timeseries Deep Learning Machine Learning Python Pytorch fastai | State-of-the-art Deep Learning library for Time Series and Sequences in Pytorch / fastai

Apache License 2.0

5.21k stars 651 forks source link

Just one small error I found

https://github.com/timeseriesAI/tsai/blob/main/tsai/analysis.py#L132

'ablation' feature importance sometimes show wrong wrong accuracy_change sign and therefore wrong order. 'accuracy_change' needs sometimes to be multiplied by -1, but not always. then sorted again by accuracy change, so its results are similar to 'permutation method' results

Thanks for the great work!

PD: Could you confirm that the most important features will be those with a lower value on its accuracy result? (since higher accuracy is better, so low accuracy when not using this feature means this feature was important).

timeseriesAI / tsai

error in permutation feature importance result (ablation method) #630