Open jloveric opened 1 month ago
Ok, I figure it out. For some reason the starting index was going beyond those defined in indices. Fixed in my fork here https://github.com/RezaKakooee/space_layout_gym/compare/SpaceLayoutGym-v0...jloveric:space_layout_gym:SpaceLayoutGym-v0#diff-af26a04a04e22cd3dd1bcf7161a6c39ac8e5347972abc6d83124d48f3d0c7437R88 Added the line
if start > len(indices) :
break
This could be entirely due to my setup and any mods to get it runningn (but also posting in case anyone else runs into it), but the initial losses are NaN due to tensors being empty. During training they have a size, but later on they are empty and so return Nan as loss (so it's not due to underflow or overflow). Maybe you've seen this issue. Trying to get to the bottom of it
and a few of the tensors being used inside the loss
initially these actually have a size during every training step.