Add a removed field to framework definitions, which stops any benchmark from being run on the framework definition and instead provide the message provided in the removed field
Remove the autoxgboost integration, and replace documentation examples referring to it by references to ranger.
Addition of removed field
The aim was to provide users with a clear message of why the framework no longer runs, and a hint as to how they could possibly still run the no longer framework. I did not want to duplicate the messages, or introduces new files, so I added a removed property which can be looked up on frameworks.yaml even if no definition of the framework exists on the respective frameworks_tag.yaml file.
The new loading behavior is specifically, try to load the tagged version and then:
if it exists and has the removed property: stop and show the removed message.
if it does not exist, but there is an untagged version in frameworks.yaml and that one has the removed property: stop and show that removed message.
otherwise, as before
This allows us to disable individual tagged definitions (by adding a removed message in only the respective framework file), but also allows us to add a single removed message for all definitions (by removing it from all files, except frameworks.yaml, and providing the error message through that file).
I noticed the loading behavior can perhaps be refactored a little (as well as error messaging), but left it as separate work item (#518).
Removal of AutoXGBoost
Calling runbenchmark.py for any definition of autoxgboost now stops with the following error message:
ERROR:
Framework definition `autoxgboost` has been removed from the benchmark: Autoxgboost is no longer supported. The last confirmed working version was at v2.0.2 with docker image 'automlbenchmark/autoxgboost:908631665f4c763d548ed203d48f05fe68613844-v2.0.2'.
PR does two things:
removedfield to framework definitions, which stops any benchmark from being run on the framework definition and instead provide the message provided in theremoved fieldranger.Addition of
removedfieldThe aim was to provide users with a clear message of why the framework no longer runs, and a hint as to how they could possibly still run the no longer framework. I did not want to duplicate the messages, or introduces new files, so I added a
removedproperty which can be looked up onframeworks.yamleven if no definition of the framework exists on the respectiveframeworks_tag.yamlfile.The new loading behavior is specifically, try to load the tagged version and then:
removedproperty: stop and show theremovedmessage.frameworks.yamland that one has theremovedproperty: stop and show thatremovedmessage.This allows us to disable individual tagged definitions (by adding a
removedmessage in only the respective framework file), but also allows us to add a singleremovedmessage for all definitions (by removing it from all files, exceptframeworks.yaml, and providing the error message through that file).I noticed the loading behavior can perhaps be refactored a little (as well as error messaging), but left it as separate work item (#518).
Removal of AutoXGBoost
Calling
runbenchmark.pyfor any definition ofautoxgboostnow stops with the following error message: