shukon
commented
4 years ago @PhMueller Hi Philipp,
good idea about the hint. But I cannot reproduce your second error. I'm getting another one, interestingly, on fanova and lpi (and an expected but annoying error on parallel coordinates, which should just skip constants altogether I guess), but initialization and everything else works fine (report attached). Can you give me your pip freeze / conda list --export output?
```
INFO:pimp.LPI:--------------------------------------------------> Found 500 valid neighbors
ERROR:cave.cavefacade.CAVE:local variable 'p' referenced before assignment
Traceback (most recent call last):
File "/home/shuki/Repos/CAVE/cave/cavefacade.py", line 55, in wrap
analyzer = f(self, *args, **kw)
File "/home/shuki/Repos/CAVE/cave/cavefacade.py", line 341, in local_parameter_importance
return LocalParameterImportance(self.runscontainer)
File "/home/shuki/Repos/CAVE/cave/analyzer/parameter_importance/local_parameter_importance.py", line 22, in __init__
self.parameter_importance("lpi")
File "/home/shuki/Repos/CAVE/cave/analyzer/parameter_importance/base_parameter_importance.py", line 60, in parameter_importance
'bokeh': components(run.pimp.evaluator.plot_bokeh(show_plot=False))
File "/home/shuki/niki/virtualenvs/cave_dev/lib/python3.6/site-packages/pimp/evaluator/local_parameter_importance.py", line 450, in plot_bokeh
plots.append(Panel(child=Row(p), title=param))
UnboundLocalError: local variable 'p' referenced before assignment
Traceback (most recent call last):
File "/home/shuki/niki/virtualenvs/cave_dev/bin/cave", line 33, in
sys.exit(load_entry_point('cave', 'console_scripts', 'cave')())
File "/home/shuki/Repos/CAVE/cave/cave_cli.py", line 295, in entry_point
cave.main_cli()
File "/home/shuki/Repos/CAVE/cave/cave_cli.py", line 290, in main_cli
cave.analyze()
File "/home/shuki/Repos/CAVE/cave/utils/timing.py", line 11, in wrap
result = f(*args, **kw)
File "/home/shuki/Repos/CAVE/cave/cavefacade.py", line 233, in analyze
self.parameter_importance(self._get_dict(self.website, "Parameter Importance"))
File "/home/shuki/Repos/CAVE/cave/cavefacade.py", line 356, in parameter_importance
self.local_parameter_importance(d=d)
File "/home/shuki/Repos/CAVE/cave/cavefacade.py", line 55, in wrap
analyzer = f(self, *args, **kw)
File "/home/shuki/Repos/CAVE/cave/cavefacade.py", line 341, in local_parameter_importance
return LocalParameterImportance(self.runscontainer)
File "/home/shuki/Repos/CAVE/cave/analyzer/parameter_importance/local_parameter_importance.py", line 22, in __init__
self.parameter_importance("lpi")
File "/home/shuki/Repos/CAVE/cave/analyzer/parameter_importance/base_parameter_importance.py", line 60, in parameter_importance
'bokeh': components(run.pimp.evaluator.plot_bokeh(show_plot=False))
File "/home/shuki/niki/virtualenvs/cave_dev/lib/python3.6/site-packages/pimp/evaluator/local_parameter_importance.py", line 450, in plot_bokeh
plots.append(Panel(child=Row(p), title=param))
UnboundLocalError: local variable 'p' referenced before assignment
```
```
INFO:fanova.visualizer.Visualizer:creating ../tmp/analysis_data/2433029771847169631/fanova/lr_batch_size.png
ERROR:cave.cavefacade.CAVE:must be str, not int
Traceback (most recent call last):
File "/home/shuki/Repos/CAVE/cave/cavefacade.py", line 55, in wrap
analyzer = f(self, *args, **kw)
File "/home/shuki/Repos/CAVE/cave/cavefacade.py", line 324, in cave_fanova
fanova = Fanova(self.runscontainer)
File "/home/shuki/Repos/CAVE/cave/analyzer/parameter_importance/fanova.py", line 35, in __init__
self.parameter_importance("fanova")
File "/home/shuki/Repos/CAVE/cave/analyzer/parameter_importance/base_parameter_importance.py", line 60, in parameter_importance
'bokeh': components(run.pimp.evaluator.plot_bokeh(show_plot=False))
File "/home/shuki/niki/virtualenvs/cave_dev/lib/python3.6/site-packages/pimp/evaluator/fanova.py", line 256, in plot_bokeh
inc_indices = [labels.index(val) for val in values]
File "/home/shuki/niki/virtualenvs/cave_dev/lib/python3.6/site-packages/pimp/evaluator/fanova.py", line 256, in
inc_indices = [labels.index(val) for val in values]
TypeError: must be str, not int
Traceback (most recent call last):
File "/home/shuki/niki/virtualenvs/cave_dev/bin/cave", line 33, in
sys.exit(load_entry_point('cave', 'console_scripts', 'cave')())
File "/home/shuki/Repos/CAVE/cave/cave_cli.py", line 295, in entry_point
cave.main_cli()
File "/home/shuki/Repos/CAVE/cave/cave_cli.py", line 290, in main_cli
cave.analyze()
File "/home/shuki/Repos/CAVE/cave/utils/timing.py", line 11, in wrap
result = f(*args, **kw)
File "/home/shuki/Repos/CAVE/cave/cavefacade.py", line 233, in analyze
self.parameter_importance(self._get_dict(self.website, "Parameter Importance"))
File "/home/shuki/Repos/CAVE/cave/cavefacade.py", line 353, in parameter_importance
self.cave_fanova(d=d)
File "/home/shuki/Repos/CAVE/cave/cavefacade.py", line 55, in wrap
analyzer = f(self, *args, **kw)
File "/home/shuki/Repos/CAVE/cave/cavefacade.py", line 324, in cave_fanova
fanova = Fanova(self.runscontainer)
File "/home/shuki/Repos/CAVE/cave/analyzer/parameter_importance/fanova.py", line 35, in __init__
self.parameter_importance("fanova")
File "/home/shuki/Repos/CAVE/cave/analyzer/parameter_importance/base_parameter_importance.py", line 60, in parameter_importance
'bokeh': components(run.pimp.evaluator.plot_bokeh(show_plot=False))
File "/home/shuki/niki/virtualenvs/cave_dev/lib/python3.6/site-packages/pimp/evaluator/fanova.py", line 256, in plot_bokeh
inc_indices = [labels.index(val) for val in values]
File "/home/shuki/niki/virtualenvs/cave_dev/lib/python3.6/site-packages/pimp/evaluator/fanova.py", line 256, in
inc_indices = [labels.index(val) for val in values]
TypeError: must be str, not int
```
It works for me with cave . --skip lpi fanova parallel_coordinates and results in this report
report.zip
<details My pip freeze>
appdirs==1.4.3
argcomplete==1.11.1
backcall==0.2.0
bokeh==1.1.0
CacheControl==0.12.6
-e git+git@github.com:automl/CAVE.git@5c16adf8a37d291482fa9d64325699bd74b4b6bb#egg=cave
certifi==2019.11.28
chardet==3.0.4
colorama==0.4.3
ConfigSpace==0.4.13
contextlib2==0.6.0
cycler==0.10.0
Cython==0.29.20
decorator==4.4.2
distlib==0.3.0
distro==1.4.0
docutils==0.16
fanova==2.0.19
hpbandster==0.7.4
html5lib==1.0.1
idna==2.8
importlib-metadata==1.7.0
ipaddr==2.2.0
ipykernel==5.3.0
ipython==7.16.1
ipython-genutils==0.2.0
jedi==0.17.1
Jinja2==2.11.2
joblib==0.15.1
jupyter-client==6.1.5
jupyter-core==4.6.3
kiwisolver==1.2.0
lazy-import==0.2.2
lockfile==0.12.2
MarkupSafe==1.1.1
matplotlib==3.2.2
msgpack==0.6.2
netifaces==0.10.9
nose==1.3.7
numpy==1.19.0
packaging==20.3
pandas==1.0.5
parso==0.7.0
patsy==0.5.1
pep517==0.8.2
pexpect==4.8.0
pickleshare==0.7.5
Pillow==7.2.0
progress==1.5
prompt-toolkit==3.0.5
psutil==5.7.0
ptyprocess==0.6.0
Pygments==2.6.1
PyImp==1.1.2
pynisher==0.5.0
pyparsing==2.4.6
pyrfr==0.8.0
Pyro4==4.80
python-dateutil==2.8.1
pytoml==0.1.21
pytz==2020.1
PyYAML==5.3.1
pyzmq==19.0.1
requests==2.22.0
retrying==1.3.3
scikit-learn==0.23.1
scipy==1.5.0
selenium==3.141.0
serpent==1.30.2
six==1.14.0
sklearn==0.0
smac==0.12.2
sobol-seq==0.2.0
statsmodels==0.11.1
threadpoolctl==2.1.0
tornado==6.0.4
tqdm==4.47.0
traitlets==4.3.3
urllib3==1.25.8
wcwidth==0.2.5
webencodings==0.5.1
zipp==3.1.0
PhMueller
Hey Joshua,
I've tried to read in some BOHB results but I stumbled across multiple obstacles.
First, CAVE (1.4.0) raised the following error.
I've remembered that you said that to read BOHB results you always need the configspace as a json. And yes I've forgotten to place the configspace.json file in the same folder. Moving it to the right place solved the error. Maybe you could add a hint in the error message above or in the documentation. :smile: (Also, I guess the file needs to be named exactly configspace.json, not config_space.json or similar. Is this correct?)
With this bug fixed, the next call raised the following error:
Maybe, you have a tip on how to solve this one. The necessary files for reproducing the error, as well as the cave log file are attached. Perhaps, this occurs because I am using constant hyperparameter in the configspace?
Cheers, Philipp
not_working.zip