Open paulguerrero opened 5 years ago
Thanks for the kinds words on Losswise
It seems the display_interval however, is over actual samples given to Losswise, not over the iteration number passed in the x parameter.
If that's a case, that's a bug. Losswise should only perform averaging when the difference in consecutive x-values is 1, eg. for the training loss:
https://github.com/Losswise/losswise-python/blob/master/losswise/__init__.py#L161
Any idea what might help resolve this bug?
I will take a harder look at the code next week regardless.
Sorry I misinterpreted the issue then. The real issue seems to be that I don't pass the x
values for my validation graph as multiples of the display_interval
. The issue is then in lines 158-159:
if self.max_iter is not None and x < self.max_iter - 1 and x % self.display_interval != 0:
return
If I read the code correctly, if the x
are not passed as multiples of the display_interval, it will be stored in tracked_value_list
, but never be added to the plot. So instead, something like this could be used:
if self.max_iter is not None and x < self.max_iter - 1 and x - self.last_plotted_x < self.display_interval:
return
First of all, thanks for this interesting alternative to Tensorboard.
One feature I am still missing is the ability to set a
display_interval
per plot line instead of per graph. In my training runs, I usually run the validation steps less frequently than training steps. It seems thedisplay_interval
however, is over actual samples given to Losswise, not over the iteration number passed in thex
parameter.So, for example, if I pass validation samples at
x
values(100, 200, 300, ...)
, and setdisplay_interval
to100
, then the validation loss plot will only update inx
value intervals of10000
instead of100
, which is longer than I would like. Loweringdisplay_interval
makes the training samples too noisy, since I pass training samples atx
values(1, 2, 3, ...)
. So the training and validation values are effectively averaged over largely different interval sizes inx
(although over the same raw number of samples).My current work-around of putting training and validation in different plots with different
display_interval
s is not ideal either, because x-axis of the plots is never quite synchronized.One way this could be solved would be to allow an option to average over same intervals in
x
instead of same number of raw samples, or an alternative could be to allow setting differentdisplay_interval
s for different plot lines in aGraph
.