search

microsoft / mwt-ds

Umbrella repository for projects related to the MWT Decision Service
187 stars 77 forks source link

Logic to create feature importance is not using the correct model #146

Open sharathmalladi opened 5 years ago

sharathmalladi commented 5 years ago

The feature importance should not use a model file that is prior to the policy reset date.

sharathmalladi commented 5 years ago

Sharing more context on the current logic: When a counterfactual evaluation is created with a start and end date, we currently use the model file that was saved on the start date as input to vw to run the feature importance: https://github.com/microsoft/mwt-ds/blob/master/DataScience/ExperimentationAzure.py#L168

This is not the right thing if the model or policy was reset in between the evaluation period and the logic needs to be tweaked to adjust for that.