search

happyharrycn / actionformer_release

Code release for ActionFormer (ECCV 2022)
MIT License
434 stars 77 forks source link

where is the undeterministic implementation？ #20

Closed juipengchang closed 2 years ago

juipengchang commented 2 years ago

If run the code directly, I will get the following error:

RuntimeError: index_addcuda does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.

tzzcl commented 2 years ago

You may directly change the includ_cuda Flag to false of fix_random_seed function in train.py.

happyharrycn commented 2 years ago

The error is likely produced by either an older version of PyTorch or an old architecture of the GPU. Setting include_cuda to False in train.py will switch to non-deterministic routines on the GPU.

juipengchang commented 2 years ago

The error is likely produced by either an older version of PyTorch or an old architecture of the GPU. Setting include_cuda to False in train.py will switch to non-deterministic routines on the GPU.

It can solve the problem but loss the reproducibility at the same time. I used PyTorch 1.8.1 and Nvidia 3090

tzzcl commented 2 years ago

We recommend to use the PyTorch version with 1.10+. In PyTorch 1.10+, it solves the reproducibility issue. You may want to switch to PyTorch 1.10+ and have another retry.

juipengchang commented 2 years ago

We recommend to use the PyTorch version with 1.10+. In PyTorch 1.10+, it solves the reproducibility issue. You may want to switch to PyTorch 1.10+ and have another retry.

OK, thank you. I will have a try.