This commit changes mean loss and accuracy calculation by keeping an accumulator for loss and accuracy, instead of lists of all previous values and summing those at each step.
The graph below shows the #trained tokens per time.
gray is with this commit and track_mem set to False
This commit changes mean loss and accuracy calculation by keeping an accumulator for loss and accuracy, instead of lists of all previous values and summing those at each step.
The graph below shows the #trained tokens per time.
step time per step