search

openml / automlbenchmark

OpenML AutoML Benchmarking Framework
https://openml.github.io/automlbenchmark
MIT License
398 stars 132 forks source link

Update to current autosklearn and SMAC versions #145

Closed flokde closed 4 years ago

flokde commented 4 years ago

Hi Seb,

I have been running in an error with using autosklearn through the benchmark. See the full error message below. Though I was running a classification task, the error message seems to share a lot with the one in this thread (using autosklearn regressor) in the auto-sklearn repository

The thread seems to suggest the bug was solved in version 0.7.0 of autosklearn, which includes an updated version of SMAC. I am not entirely sure of course that this is the source of the error I am experiencing (happy to be pointed towards any mistakes I may have made), but it might be worthwhile to update to autosklearn 0.7.0 in any case.

I am running a highly imbalanced binary classifications ask on tabular data (about 2.3e6 data points) on CentOS 7.8.2003. The virtual environment is set up as per the suggestions in your README/HOWTO files, the autosklearn venv is unchanged.

Thanks for all your great work! All the best, Florian

File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap

self.run()

File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run

self._target(*self._args, **self._kwargs)

File "/home/krl/projects/automlbenchmark/frameworks/autosklearn/venv/lib64/python3.6/site-packages/autosklearn/estimators.py", line 15, in _fit_automl

return automl.fit(load_models=load_models, **kwargs)

File "/home/krl/projects/automlbenchmark/frameworks/autosklearn/venv/lib64/python3.6/site-packages/autosklearn/automl.py", line 996, in fit

load_models=load_models,

File "/home/krl/projects/automlbenchmark/frameworks/autosklearn/venv/lib64/python3.6/site-packages/autosklearn/automl.py", line 208, in fit

only_return_configuration_space=only_return_configuration_space,

File "/home/krl/projects/automlbenchmark/frameworks/autosklearn/venv/lib64/python3.6/site-packages/autosklearn/automl.py", line 489, in _fit

_proc_smac.run_smbo()

File "/home/krl/projects/automlbenchmark/frameworks/autosklearn/venv/lib64/python3.6/site-packages/autosklearn/smbo.py", line 513, in run_smbo

logger=smac.solver.logger,

File "/home/krl/projects/automlbenchmark/frameworks/autosklearn/venv/lib64/python3.6/site-packages/smac/optimizer/pSMAC.py", line 51, in read

configuration_space)

File "/home/krl/projects/automlbenchmark/frameworks/autosklearn/venv/lib64/python3.6/site-packages/smac/runhistory/runhistory.py", line 381, in update_from_json

new_runhistory.load_json(fn, cs)

File "/home/krl/projects/automlbenchmark/frameworks/autosklearn/venv/lib64/python3.6/site-packages/smac/runhistory/runhistory.py", line 348, in load_json

all_data = json.load(fp, object_hook=StatusType.enum_hook)

File "/usr/lib64/python3.6/json/init.py", line 299, in load

parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)

File "/usr/lib64/python3.6/json/init.py", line 367, in loads

return cls(**kw).decode(s)

File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode

obj, end = self.raw_decode(s, idx=_w(s, 0).end())

File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode

raise JSONDecodeError("Expecting value", s, err.value) from None

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

sebhrusen commented 4 years ago

Hi @flokde

Thanks for reporting this issue: I already have a PR that should also partly solve those kind of issues in the future, see https://github.com/openml/automlbenchmark/pull/144

That PR also includes an upgrade of autosklearn to 0.7.0, and more importantly, it will allow users to test newer or older versions by themselves in a purely declarative way (except if major changes in the setup process or requirements).

I still need to test all those changes, but should be available very soon: feel free to try it now.

flokde commented 4 years ago

That's great to hear, thank you! So if I wanted to try out autoskelarn 0.7.0 through the benchmark, I could play around with the code in your PR? And either way these changes are gonna be merged to the stable version soon? Thanks again and all the best, Florian

sebhrusen commented 4 years ago

@flokde FYI, this is fixed with #144. Sorry for late reply, didn't have time to spend on this lately. autosklearn is now set to 0.8.0 by default, see example in https://github.com/openml/automlbenchmark/blob/master/docs/HOWTO.md#framework-definition if you want to switch easily between framework versions.

Also note that I haven't rebuilt any docker images yet: if you want to use a new version with docker, you'll have to build the image by yourself.