Open TMorville opened 4 years ago
Is this still an issue?
It is. However, I think my baseline might be wrong.
I think the relevant baseline to compare performance to should be a bayesian regression trained directly on the data, instead of the output from the neural network. Do you agree?
I have been playing around with the DCBTrainer and found some potential inconsistencies.
1) StatlogData example found here
and code to evaluate
2) WineDataBandit example found here
Define the bandit
training
and evaluation
For both cases (and the third titanic case referenced in #301 ) when training both reward and regret increases which could point to no actual learning happening and the increase in reward comes purely from randomly guessing instead of learning.
Notice that for the Statlog data, the evaluation col is the last column, while in the wine data, its the first column.