Generated predictions and evaluation metrics should be traceable to the parameters they were created with, i.e. model, training data and test data.
This information is stored in the files train_config.toml and test_config.toml. What we need is a way to link the predictions and evaluation results to the config files they are based on.
In the simplest form, each row in the metrics file (e.g. called eval.jsonl) would include a reference to the test_config.toml file that it is linked to. (This assumes that the model directory referenced in test_config.toml contains in turn a copy of train_config.toml at the time of training. Thus we have linked both the training and the test parameters to the evaluation results.) A row in the metrics file then would look like this:
Here we would have a directory that serves as an archive, where each test_config.toml that was ever used is stored under a unique number. Thus, unless we hide the archived configs from git (and thereby do not save them or make them redistributable), we have to make a new git commit for each evaluation we run. To create the unique filename, run:
filename=`md5sum test_config.toml | head -c 8`.toml
cp test_config.toml hidden/test_configs/$filename
ybracke
commented
10 months ago
Generated predictions and evaluation metrics should be traceable to the parameters they were created with, i.e. model, training data and test data.
This information is stored in the files
train_config.tomlandtest_config.toml. What we need is a way to link the predictions and evaluation results to the config files they are based on.In the simplest form, each row in the metrics file (e.g. called
eval.jsonl) would include a reference to thetest_config.tomlfile that it is linked to. (This assumes that themodeldirectory referenced intest_config.tomlcontains in turn a copy oftrain_config.tomlat the time of training. Thus we have linked both the training and the test parameters to the evaluation results.) A row in the metrics file then would look like this:Here we would have a directory that serves as an archive, where each
test_config.tomlthat was ever used is stored under a unique number. Thus, unless we hide the archived configs from git (and thereby do not save them or make them redistributable), we have to make a new git commit for each evaluation we run. To create the unique filename, run: