Open fvisin opened 8 years ago
I've never heard of this one. Might be related to progressbar2 which I've never tried with smartdispatch. I don't know if @MarcCote knows more about this issue. For now, can you tell us on which cluster you've seen this issue and do you have code to reproduce?
I was using Helios. You can almost certaintly reproduce it with this code (I'm using the same main loop, which is where the progressbar is called): https://github.com/fvisin/reseg
Thank you for investigating the issue!
The logs of my jobs are missing some lines (I print one line every batch, approx 6 out of 7 are missing!) and sometimes the lines are scrambled (the outputs of different epochs are interleaved). Note that these are two separate issues, i.e., I checked that the missing lines are not just somewhere else in the log, they are actually missing.
It might have to do with progressbar2, but I have no evidence of this. The only reason for this claim is that there might be an incompatibility with progressbar rewriting multiple times the same line.