problems with predict_marginalized_over_instances

ndangtt commented 5 years ago

Firstly, my apology if this is a false alarm. I have some issues running the examples, and the following is my guess when looking at the code. However, it could be that I did something incorrectly! :)

I tried the following command in examples/spear_qcp

pimp -S smac3-output_2017-04-03_17:13:13/scenario.txt -H smac3-output_2017-04-03_17:13:13/runhistory.json -M forward-selection

And I get the following output:

INFO:Importance:#RunHistories found: 1
INFO:Importance:Combined number of Runhistory data points: 8
INFO:Importance:Number of Configurations: 3
INFO:Importance:Converting Data and constructing Model
INFO:Importance:Using model <pimp.epm.unlogged_epar_x_rfwi.UnloggedEPARXrfi object at 0x7f932f1d6208>
INFO:Importance:Size of training X: (8, 80)
INFO:Importance:Size of training y: (8, 1)
INFO:Importance:Data was not imputed
INFO:Importance:Thus the size of X might be smaller than the datapoints in the RunHistory
INFO:Importance:Fitting Model
INFO:Importance:Reading traj_file: /home/nttd/Dropbox/St-Andrews/irace-project/ParameterImportance/examples/spear_qcp/smac3-output_2017-04-03_17:13:13/traj_aclib2.json
Traceback (most recent call last):
  File "/home/nttd/Dropbox/softwares/miniconda3/bin/pimp", line 11, in <module>
    load_entry_point('PyImp==1.0.5', 'console_scripts', 'pimp')()
  File "/home/nttd/Dropbox/softwares/miniconda3/lib/python3.6/site-packages/PyImp-1.0.5-py3.6.egg/pimp/pimp.py", line 262, in cmd_line_call
    forwardsel_cv=args.forwardsel_cv)  # create importance object
  File "/home/nttd/Dropbox/softwares/miniconda3/lib/python3.6/site-packages/PyImp-1.0.5-py3.6.egg/pimp/importance/importance.py", line 90, in __init__
    self._load_incumbent(traj_file, runhistory_file, incumbent)
  File "/home/nttd/Dropbox/softwares/miniconda3/lib/python3.6/site-packages/PyImp-1.0.5-py3.6.egg/pimp/importance/importance.py", line 190, in _load_incumbent
    np.array([impute_inactive_values(incumbents[-1][0]).get_array()])))
  File "/home/nttd/Dropbox/softwares/miniconda3/lib/python3.6/site-packages/smac/epm/base_epm.py", line 242, in predict_marginalized_over_instances
    raise ValueError('Rows in X should have %d entries but have %d!' % (len(self.types), X.shape[1]))
ValueError: Rows in X should have 80 entries but have 26!

It seems to me that there's an issue with the function predict_marginalized_over_instances in SMAC3's code. More specifically these lines:

if X.shape[1] != len(self.types):
    raise ValueError('Rows in X should have %d entries but have %d!' % (len(self.types), X.shape[1]))

As X contains only algorithm parameters, while self.types contains both algorithm parameters and instance features. When I disable this if, it seems to run fine.

Could you please help to check it?

Thank you very much! Nguyen

AndreBiedenkapp commented 5 years ago

Hi, sorry for the late reply. I have a feeling that this might be caused by a "too new" SMAC version. PyImp currently is a bit lacking behind SMAC and might not be fully compatible.

ndangtt commented 5 years ago

Thanks for your reply!

Looking at the history of that file in SMAC, it was added on Nov 16, 2018.

So I made a "fresh" installation of pimp by removing everything currently installed, and do pip install pyimp, and get that same error when running the example.

It seems to me that this might be a bug in the current SMAC code. I'll post an issue there.

automl / ParameterImportance

problems with predict_marginalized_over_instances #105