G-Research / fasttrackml

Experiment tracking server focused on speed and scalability
https://fasttrackml.io/
Apache License 2.0
96 stars 18 forks source link

Metric.Iter logging incorrectly #1328

Open suprjinx opened 1 week ago

suprjinx commented 1 week ago

From the community slack channel:

Image

suprjinx commented 1 week ago

It seems like one potential solution is to recalculate the iter values after each batch is finished -- using SQL so that other threads are integrated properly. Order by step and timestamp to produce the ordering.