search

automl / HPOlib1.5

GNU General Public License v3.0
69 stars 27 forks source link

Benchmark version #45

Open KEggensperger opened 7 years ago

KEggensperger commented 7 years ago

How should we handle different versions of our benchmarks?

For example, if we change a configurationspace, users should be able to see that this space is different from something that was previously used.

I propose to add "version" to the metadictionary. For benchmarks such as autosklearn we use the version of that package, for other benchmarks we manually maintain the version number?

mfeurer commented 7 years ago

I don't making the internal benchmark version equal to the version number of the external software is a good idea. This wouldn't allow us to fix bugs/change minor things unless the external software changes.

Besides that, yes, a great idea!