What do you think of this DSL to formalize experiment evaluation protocols? As you can see, it's optional... all the tasks & views & splits etc. that we've been talking about are still there unmodified, there's just a "dsl" attribute at the top level that formally specifies what the user is supposed to do. Maybe for performance reasons the user might not want to implement is learning algorithm as a visitor (as I showed in test_dslang) but at least for unit testing purposes this specifies exactly what the right answer is, even if the user wants to compute it some other way.
@npinto @poilvert
What do you think of this DSL to formalize experiment evaluation protocols? As you can see, it's optional... all the tasks & views & splits etc. that we've been talking about are still there unmodified, there's just a "dsl" attribute at the top level that formally specifies what the user is supposed to do. Maybe for performance reasons the user might not want to implement is learning algorithm as a visitor (as I showed in test_dslang) but at least for unit testing purposes this specifies exactly what the right answer is, even if the user wants to compute it some other way.