search

byu-dml / d3m-experimenter

A distributed system for creating, running, and persisting many machine learning experiments.
0 stars 0 forks source link

Update dataset doc and problem doc getters to use D3M utils #92

Closed e13h closed 3 years ago

e13h commented 3 years ago

In this PR, I updated utils.py to include global variables that hold the mappings of dataset ids and problem ids to corresponding file paths. Since the variables take some time to create (the D3M code has to scan all the datasets), they are actually only filled the first time you call either get_dataset_doc_path() or get_problem_path(). That way, if you just import experimenter.utils you don't have to wait around for these global variables to get filled if you don't even call these update getters. All subsequent calls are very fast since the dictionaries are now cached.

Closes #91

bjschoenfeld commented 3 years ago

can you add a couple basic tests to make sure this code works. use the datasets directory found in this repo.