Currently the database readers and database writers use an automagical get_experiment_name function to find the results path. This does not work in testing: when DefinitionPart() is called through pytest, the experiment name is "bin". However, when ExperimentDefinition() is called in the user's experiment.py file, the experiment name is identified correctly. As a result, all tests involving metadata readers and writers are forced to implement stubbed classes as below, and it is impossible to directly test whether the original (non-stubbed) metadata readers/writers are working as intended.
Example:
Both tests/test_ExperimentDefinition.py and tests/acceptance/test_softmax.py use janky extra code to pass the results path around.
Proposed solution:
Add self.exp_name and self.get_results_path() to all the database readers/writers, as above. Currently we would need to change ExperimentDefinition, DefinitionPart, and Scheduler.
Currently the database readers and database writers use an automagical
get_experiment_name
function to find the results path. This does not work in testing: whenDefinitionPart()
is called through pytest, the experiment name is "bin". However, whenExperimentDefinition()
is called in the user'sexperiment.py
file, the experiment name is identified correctly. As a result, all tests involving metadata readers and writers are forced to implement stubbed classes as below, and it is impossible to directly test whether the original (non-stubbed) metadata readers/writers are working as intended.Example: Both
tests/test_ExperimentDefinition.py
andtests/acceptance/test_softmax.py
use janky extra code to pass the results path around.Proposed solution: Add
self.exp_name
andself.get_results_path()
to all the database readers/writers, as above. Currently we would need to change ExperimentDefinition, DefinitionPart, and Scheduler.