rswc
commented
9 months ago Maybe we can do this later, as a dedicated and separate experiment? I suggest after we're done playing around with model performance (as in, accuracy and stuff). I feel like adding this now wouldn't be worth the effort and compute.
As we may want to include some discussion on the methods' performance, tracking the time taken at each step would be beneficial.
We know wandb collects some system metrics out of the box — are those satisfactory? What about comparing methods which run in batch mode with those that don't?