Overflow error for ablation

[robintibor]

So for same configuration as in #9 I now have a different error when using ablation modus:

$ python3.5 /home/schirrmr/programs/ParameterImportance/scripts/evaluate.py --scenario_file hybrid_scenario.txt --history '.runhistory*.json' --modus ablation --trajectory traj_aclib2.json

/home/schirrmr/.local/lib/python3.5/site-packages/matplotlib/__init__.py:1401: UserWarning:  This call to matplotlib.use() has no effect
because the backend has already been chosen;
matplotlib.use() must be called *before* pylab, matplotlib.pyplot,
or matplotlib.backends is imported for the first time.

  warnings.warn(_use_error_msg)
INFO:Importance:Reading Scenario file and files specified in the scenario
INFO:scenario:Reading scenario file: hybrid_scenario.txt
INFO:scenario:Output to hybrid-results
INFO:Importance:Reading Runhistory
INFO:Importance:Combined number of Runhistory data points: 656
INFO:Importance:Converting Data and constructing Model
INFO:Importance:Setting up Evaluation Method
INFO:Importance:Running evaluation method Ablation
/home/schirrmr/programs/ParameterImportance/pimp/epm/unlogged_rf_with_instances.py:92: RuntimeWarning: overflow encountered in power
  tmpx = np.array(list(map(lambda x_: np.power(10, x_), self.rf.all_leaf_values(x))))  # unlog values
/home/schirrmr/.local/lib/python3.5/site-packages/numpy/core/_methods.py:116: RuntimeWarning: overflow encountered in multiply
  x = um.multiply(x, x, out=x)
/home/schirrmr/.local/lib/python3.5/site-packages/numpy/core/_methods.py:112: RuntimeWarning: invalid value encountered in subtract
  x = asanyarray(arr - arrmean)
/home/schirrmr/programs/ParameterImportance/pimp/epm/unlogged_rf_with_instances.py:146: RuntimeWarning: invalid value encountered in true_divide
  a=(-np.inf - mean_var[0][p]) / np.sqrt(mean_var[1][p]),
CRITICAL:RF:Prediction is not finite cdf 0.5, lower_pred nan; Setting nan to 1e+10
[...]
Traceback (most recent call last):
  File "/home/schirrmr/programs/ParameterImportance/scripts/evaluate.py", line 33, in <module>
    importance_value_dict = importance.evaluate_scenario(args.modus)
  File "/home/schirrmr/programs/ParameterImportance/pimp/importance/importance.py", line 244, in evaluate_scenario
    return self.evaluator.run()
  File "/home/schirrmr/programs/ParameterImportance/pimp/evaluator/ablation.py", line 263, in run
    best_performance = round_performances[best_idx]  # greedy choice of parameter to fix
IndexError: list index out of range

My costs that I am minimizing can range from 0 to about 500 is this a problem?

[AndreBiedenkapp]

This is probably caused by the very large cutoff you used in order to get pimp running and thus not happen again after fixing issue #9. Just to be on the save side I'll handle this error more appropriately.

[robintibor]

Hm I get the same error if I set

cutoff_time = 1

:/

[AndreBiedenkapp]

With the previous commit, I was not able to reproduce the error using a simple branin example (which I also uploaded). I suppose this error happened because the wrong model was used wich did some log transformations intended for scenarios with running-time as objective.

The latest commit just added an assert that will result in a more meaningful Failure message than an Index Error

[AndreBiedenkapp]

@robintibor Let me know if you still experience this error, or if you now get 'No improving parameter found!' when you run your scenario evaluations (for wich you don't need the cutoff anymore)

[robintibor]

It seems fine now! :) Thanks :) Now I just have to understand all the outputs and plots ;)