Additional analysis for paper

[stevehadd]

Based on discussion with Francesco, here are some additional things we could try for additional analysis to add to the paper to make a stronger case for how we have chosen to do things:

• cal;culate permutation importance to show which features are useful
• include balanced accuracy and some other metrics that are better suited to an imbalanced class problem than recall
• calculate some confusion matrices to shed more light on performance for different classes
• do some bootstrapping to get better performance for small classes through oversampling in the train set.

[stevehadd]

scikit learn bootstapping as part of cross validation: https://ogrisel.github.io/scikit-learn.org/sklearn-tutorial/modules/generated/sklearn.cross_validation.Bootstrap.html

alternatively, the pandas sample method could be used for bootstrapping. https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.sample.html

We could use this to even out class support for a variety of features:

• instrument type
• country
• year

[stevehadd]

Some more ways to look at links between features https://datascience.stackexchange.com/questions/893/how-to-get-correlation-between-two-categorical-variable-and-a-categorical-variab

[stevehadd]

various ways to do feature selection more systematically using sklearn or pandas:

• https://towardsdatascience.com/feature-selection-with-pandas-e3690ad8504b
• https://scikit-learn.org/stable/modules/feature_selection.html

[stevehadd]

Permutation feature importance algorithm implementation: https://scikit-learn.org/stable/modules/permutation_importance.html

[stevehadd]

Docs for more metrics:

• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.balanced_accuracy_score.html
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html

[stevehadd]

This has been implemeted in various notebooks and updated for the batch code in PR #111.