panahiparham / ml-experiment-definition

MIT License
2 stars 3 forks source link

Can't set experiment name in pytest #18

Closed yasuiniko closed 3 weeks ago

yasuiniko commented 1 month ago

Currently the database readers and database writers use an automagical get_experiment_name function to find the results path. This does not work in testing: when DefinitionPart() is called through pytest, the experiment name is "bin". However, when ExperimentDefinition() is called in the user's experiment.py file, the experiment name is identified correctly. As a result, all tests involving metadata readers and writers are forced to implement stubbed classes as below, and it is impossible to directly test whether the original (non-stubbed) metadata readers/writers are working as intended.

Example: Both tests/test_ExperimentDefinition.py and tests/acceptance/test_softmax.py use janky extra code to pass the results path around.

class stubbed_ExperimentDefinition(ExperimentDefinition):
    def __init__(self, exp_name: str, part_name: str, version: int, base: str | None = None):
        self.exp_name = exp_name
        super().__init__(part_name, version, base)

    def get_results_path(self, base_path) -> str:
        return os.path.join(base_path, 'results', self.exp_name)

Proposed solution: Add self.exp_name and self.get_results_path() to all the database readers/writers, as above. Currently we would need to change ExperimentDefinition, DefinitionPart, and Scheduler.