genforce / insgen

[NeurIPS 2021] Data-Efficient Instance Generation from Instance Discrimination
https://genforce.github.io/insgen/
Other
101 stars 4 forks source link

Error while resuming training! #7

Closed 49xxy closed 1 year ago

49xxy commented 1 year ago

Thank you for your excellent job! When I was used " --resume" to resume training, I received annoying error: Constructing networks... Resuming from "/content/drive/MyDrive/FakeCLR-main/training-runs-7k/00002-pv-lianpu7k-paper256-gamma10-kimg5000-batch8/network-snapshot-000200.pkl" Traceback (most recent call last): File "train.py", line 583, in <module> main() # pylint: disable=no-value-for-parameter File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "train.py", line 576, in main subprocess_fn(rank=0, args=args, temp_dir=temp_dir) File "train.py", line 421, in subprocess_fn training_loop.training_loop(rank=rank, **args) File "/content/drive/MyDrive/FakeCLR-main/training/training_loop.py", line 172, in training_loop misc.copy_params_and_buffers(resume_data[name], module, require_all=False) File "/content/drive/MyDrive/FakeCLR-main/torch_utils/misc.py", line 160, in copy_params_and_buffers tensor.copy_(src_tensors[name].detach()).requires_grad_(tensor.requires_grad) RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

Sincerely hope to get your help!