samjmolyneux / eppi-text-classification

Classifying papers by their abstracts and titles.
2 stars 0 forks source link

Add limits that cleverly enforce the size of datasets that can be used #45

Closed samjmolyneux closed 1 month ago

samjmolyneux commented 2 months ago

Both hyperparameter optimisation and shap values stop working properly when data gets too large.

Need to figure out why for both and find a way to do clever validation.

samjmolyneux commented 2 months ago

Surely it doesnt make sense that we are having these problems with the Shap plotter, given that it supposedly can deal with massive datasets.