I'm trying to run PipeDream_2BW branch without--recompute_step
However, I encountered the following Exception.
('one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1024]] is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!',)
I think it was related to the weights versions (2 versions for PipeDream_2BW) stashing but I'm not sure. Same issue also happen on PipeDream on this branch if I tested it without --recompute_step . Where is the inplace operation from? Is there any clue to solve this problem?
Hi,
I'm trying to run PipeDream_2BW branch without
--recompute_stepHowever, I encountered the following Exception.('one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1024]] is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!',)I think it was related to the weights versions (2 versions for PipeDream_2BW) stashing but I'm not sure. Same issue also happen on PipeDream on this branch if I tested it without
--recompute_step. Where is the inplace operation from? Is there any clue to solve this problem?Thanks and Best Regards, Shigang