Open kosmitive opened 11 months ago
@kosmitive Do you have a real use case in mind that you could share? It will help understand the issue and context better, think of possible workarounds, and prioritize appropriately.
@dberenbaum I updated the issue description above.
@kosmitive Do you have a real use case in mind that you could share? It will help understand the issue and context better, think of possible workarounds, and prioritize appropriately.
Workaround would be to unroll the nested operator in dvc.yaml
into
stages:
dispatch_experiment_experiment_a:
matrix:
metric: ${experiments.experiment_a.metrics}
cmd: python -m scripts.dispatch_experiment
--experiment-name experiment_a
--metric-name ${item.metric}
outs:
- experiment_a/${item.metric}.csv
dispatch_experiment_experiment_b:
matrix:
metric: ${experiments.experiment_b.metrics}
cmd: python -m scripts.dispatch_experiment
--experiment-name experiment_b
--metric-name ${item.metric}
outs:
- experiment_b/${item.metric}.csv
Makes sense @kosmitive, thanks! I don't fully get how you are using metrics
in your example. Where do you use idx
and kwargs
?
@dberenbaum idx
and kwargs
are used as in
def prepare_metric(experiment_name: str, metric_name: str) -> partial:
metric_name, metric_config in params["experiments"][experiment_name]["metrics][metric_name]
metric_idx = metric_config["idx"] # <---------------------|
metric_kwargs = metric_config["kwargs"] # <---------------------| Parameters `idx` and `kwargs`
metric_cls = MetricRegistry[metric_idx]
return partial(metric_cls, **metric_kwargs)
to prepare the metrics such that they provide a unified interface. MetricRegistry
is a dictionary holding the default constructors for the different registered metrics. In that case idx
(or we could call it also metric_fn_id
) is used to access the constructor in the registry.
And do you want to run all matrix combinations in one run or only select combinations at a time?
The metrics should be managed by dvc
. Reason for that is, if I there are changes to the plotting stage the results should be reused (as itr takes up to 5 min for evaluating a metric). Imagine adding a new metric, in that case the other metrics shouldn't be recalculated.
What about at the experiment level. Do you run all experiments in a single run or do you want to invoke only experiment_a
at one time?
On all experiments. The experiment level is a good point, currently we use that pattern https://github.com/iterative/dvc/issues/9948#issuecomment-1728188440.
Regarding integrating it completely into dispatch_experiment.py
: dvc
creates a folder experiment_a/metrics
. The first run executes only metric_1
. The second run executes metric_1
and metric_2
. One run is one call of dvc exp run
. As dvc
removes the files in experiment_a/metrics
we can't rely on existing data in that folder. What do you think would be the best solution to do it on an experiment level (with caching and reusing values)?
Sometimes evaluating a certain metric calculations can take up some time. So we want to cache these results using
dvc
. For that we consider the followingparam.yaml
filecombined with the
dvc.yaml
filein order to make the looping of matrix notation more flexible.