Right now, the supervisor sends one evaluation request per (model, interval) pair. However, often, we want to evaluate the same model on many intervals. Since every gRPC request implies that the evaluator downloads the model, instantiates the model on the GPU, creates processes etc, it might make sense to instead send requests of the form (model, [list of intervals]). Then, the evaluator can instantiate one model instance, only needs to spawn a single process etc.
This reduces the overhead of evaluation in cases where the evaluation time is dominated by the orchestration stuff (e.g. yearbook) instead of forward passes. Needs careful handling at the evaluator, since we now need to reset the state of the Pytorch Evaluator per interval, and need to return the numbers correctly, but is definitly notable and should be implementable.
Right now, the supervisor sends one evaluation request per (model, interval) pair. However, often, we want to evaluate the same model on many intervals. Since every gRPC request implies that the evaluator downloads the model, instantiates the model on the GPU, creates processes etc, it might make sense to instead send requests of the form (model, [list of intervals]). Then, the evaluator can instantiate one model instance, only needs to spawn a single process etc.
This reduces the overhead of evaluation in cases where the evaluation time is dominated by the orchestration stuff (e.g. yearbook) instead of forward passes. Needs careful handling at the evaluator, since we now need to reset the state of the Pytorch Evaluator per interval, and need to return the numbers correctly, but is definitly notable and should be implementable.