search

automl / ParameterImportance

Parameter Importance Analysis Tool
http://www.ml4aad.org/
BSD 3-Clause "New" or "Revised" License
76 stars 19 forks source link

PIMP / fANOVA / Integer Parameter does not seem to work well #32

Closed janvanrijn closed 7 years ago

janvanrijn commented 7 years ago

All integer parameter plots are 'constant'. I can give a MWE later. If I directly run through fANOVA it works correctly.

Krxsy commented 7 years ago

Ok, I think I found out why (hypercube is not taken into account). I'll fix it today :-)

Krxsy commented 7 years ago

It should work now. Let me know if it still happens.

KEggensperger commented 7 years ago

Hi, unfortunately I think this is not yet fixed. I used the following command to produce fanova results resulting in empty plots for some hyperparameters.

python <path/to/pimp/repo>/ParameterImportance/scripts/evaluate.py -S scenario.txt -M fanova -H "*/runhistory.json"

I would guess that integer hyperparameters on log scale and categorical hyperparameters with strings ["1", "2", "3", etc] are still a problem.

The plots and data to reproduce this is here: probsat-7sat90_pimped.tar.gz

PS: This is not urgent for me.

AndreBiedenkapp commented 7 years ago

@Krxsy I just tested everything with the latest fANOVA and this problem still exists. I was just checking in to see which issues I can/have to work on

Krxsy commented 7 years ago

Both issues are fixed and it should work now.