search

ltkong218 / IFRNet

IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation (CVPR 2022)
MIT License
276 stars 23 forks source link

The problem encountered in training procedure #10

Closed NK-CS-ZZL closed 2 years ago

NK-CS-ZZL commented 2 years ago

I attempt to follow this excellent work and encounter some problems in training. I'll appreciate it if you could help me solve it. The log is presented below:

2022-07-09 13:40:33:INFO:Namespace(batch_size=6, device=device(type='cuda', index=0), epochs=300, eval_interval=1, local_rank=0, log_path='checkpoint/IFRNet', lr_end=1e-05, lr_start=0.0001, model_name='IFRNet', num_workers=6, resume_epoch=0, resume_path=None, world_size=4) Distributed Data Parallel Training IFRNet on Rank 2 Traceback (most recent call last): File "train_vimeo90k.py", line 188, in train(args, ddp_model) File "train_vimeo90k.py", line 89, in train loss.backward() File "/home/vm411/miniconda3/envs/ifrnet/lib/python3.7/site-packages/torch/_tensor.py", line 363, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/home/vm411/miniconda3/envs/ifrnet/lib/python3.7/site-packages/torch/autograd/init.py", line 175, in backward allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [6, 32, 112, 112]], which is output 0 of AsStridedBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True). Traceback (most recent call last): File "train_vimeo90k.py", line 188, in train(args, ddp_model) File "train_vimeo90k.py", line 89, in train loss.backward() File "/home/vm411/miniconda3/envs/ifrnet/lib/python3.7/site-packages/torch/_tensor.py", line 363, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/home/vm411/miniconda3/envs/ifrnet/lib/python3.7/site-packages/torch/autograd/init.py", line 175, in backward allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [6, 32, 112, 112]], which is output 0 of AsStridedBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True). Traceback (most recent call last): File "train_vimeo90k.py", line 188, in train(args, ddp_model) File "train_vimeo90k.py", line 89, in train loss.backward() File "/home/vm411/miniconda3/envs/ifrnet/lib/python3.7/site-packages/torch/_tensor.py", line 363, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/home/vm411/miniconda3/envs/ifrnet/lib/python3.7/site-packages/torch/autograd/init.py", line 175, in backward allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [6, 32, 112, 112]], which is output 0 of AsStridedBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True). Traceback (most recent call last): File "train_vimeo90k.py", line 188, in train(args, ddp_model) File "train_vimeo90k.py", line 89, in train loss.backward() File "/home/vm411/miniconda3/envs/ifrnet/lib/python3.7/site-packages/torch/_tensor.py", line 363, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/home/vm411/miniconda3/envs/ifrnet/lib/python3.7/site-packages/torch/autograd/init.py", line 175, in backward allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [6, 32, 112, 112]], which is output 0 of AsStridedBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True). ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 29114) of binary: /home/vm411/miniconda3/envs/ifrnet/bin/python Traceback (most recent call last): File "/home/vm411/miniconda3/envs/ifrnet/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/vm411/miniconda3/envs/ifrnet/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/vm411/miniconda3/envs/ifrnet/lib/python3.7/site-packages/torch/distributed/launch.py", line 193, in main() File "/home/vm411/miniconda3/envs/ifrnet/lib/python3.7/site-packages/torch/distributed/launch.py", line 189, in main launch(args) File "/home/vm411/miniconda3/envs/ifrnet/lib/python3.7/site-packages/torch/distributed/launch.py", line 174, in launch run(args) File "/home/vm411/miniconda3/envs/ifrnet/lib/python3.7/site-packages/torch/distributed/run.py", line 718, in run )(*cmd_args) File "/home/vm411/miniconda3/envs/ifrnet/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 131, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/home/vm411/miniconda3/envs/ifrnet/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 247, in launch_agent failures=result.failures, torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

train_vimeo90k.py FAILED

And here is our conda environment:

_libgcc_mutex 0.1 main defaults _openmp_mutex 5.1 1_gnu defaults absl-py 0.15.0 pyhd3eb1b0_0 defaults aiohttp 3.8.1 py37h7f8727e_1 defaults aiosignal 1.2.0 pyhd3eb1b0_0 defaults async-timeout 4.0.1 pyhd3eb1b0_0 defaults asynctest 0.13.0 py_0 defaults attrs 21.4.0 pyhd3eb1b0_0 defaults blas 1.0 mkl defaults blinker 1.4 py37h06a4308_0 defaults brotli 1.0.9 he6710b0_2 defaults brotlipy 0.7.0 py37h27cfd23_1003 defaults bzip2 1.0.8 h7b6447c_0 defaults c-ares 1.18.1 h7f8727e_0 defaults ca-certificates 2022.4.26 h06a4308_0 defaults cachetools 4.2.2 pyhd3eb1b0_0 defaults cairo 1.16.0 h19f5f5c_2 defaults certifi 2022.6.15 py37h06a4308_0 defaults cffi 1.15.0 py37hd667e15_1 defaults charset-normalizer 2.0.4 pyhd3eb1b0_0 defaults click 8.0.4 py37h06a4308_0 defaults cryptography 3.4.8 py37hd23ed53_0 defaults cudatoolkit 11.3.1 h2bc3f7f_2 defaults cupy-cuda114 10.6.0 pypi_0 pypi cycler 0.11.0 pyhd3eb1b0_0 defaults dataclasses 0.8 pyh6d0b6a4_7 defaults dbus 1.13.18 hb2f20db_0 defaults expat 2.4.4 h295c915_0 defaults fastrlock 0.8 pypi_0 pypi ffmpeg 4.0 hcdf2ecd_0 defaults fontconfig 2.13.1 h6c09931_0 defaults fonttools 4.25.0 pyhd3eb1b0_0 defaults freeglut 3.0.0 hf484d3e_5 defaults freetype 2.11.0 h70c0345_0 defaults frozenlist 1.2.0 py37h7f8727e_0 defaults giflib 5.2.1 h7b6447c_0 defaults glib 2.69.1 h4ff587b_1 defaults google-auth 2.6.0 pyhd3eb1b0_0 defaults google-auth-oauthlib 0.4.1 py_2 defaults graphite2 1.3.14 h295c915_1 defaults grpcio 1.42.0 py37hce63b2e_0 defaults gst-plugins-base 1.14.0 h8213a91_2 defaults gstreamer 1.14.0 h28cd5cc_2 defaults harfbuzz 1.8.8 hffaf4a1_0 defaults hdf5 1.10.2 hba1933b_1 defaults icu 58.2 he6710b0_3 defaults idna 3.3 pyhd3eb1b0_0 defaults imageio 2.9.0 pyhd3eb1b0_0 defaults importlib-metadata 4.11.3 py37h06a4308_0 defaults intel-openmp 2021.4.0 h06a4308_3561 defaults jasper 2.0.14 hd8c5072_2 defaults jpeg 9e h7f8727e_0 defaults kiwisolver 1.4.2 py37h295c915_0 defaults lcms2 2.12 h3be6417_0 defaults ld_impl_linux-64 2.38 h1181459_1 defaults libffi 3.3 he6710b0_2 defaults libgcc-ng 11.2.0 h1234567_1 defaults libgfortran-ng 7.5.0 ha8ba4b0_17 defaults libgfortran4 7.5.0 ha8ba4b0_17 defaults libglu 9.0.0 hf484d3e_1 defaults libgomp 11.2.0 h1234567_1 defaults libopencv 3.4.2 hb342d67_1 defaults libopus 1.3.1 h7b6447c_0 defaults libpng 1.6.37 hbc83047_0 defaults libprotobuf 3.20.1 h4ff587b_0 defaults libstdcxx-ng 11.2.0 h1234567_1 defaults libtiff 4.2.0 h2818925_1 defaults libuuid 1.0.3 h7f8727e_2 defaults libuv 1.40.0 h7b6447c_0 defaults libvpx 1.7.0 h439df22_0 defaults libwebp 1.2.2 h55f646e_0 defaults libwebp-base 1.2.2 h7f8727e_0 defaults libxcb 1.15 h7f8727e_0 defaults libxml2 2.9.14 h74e7548_0 defaults lz4-c 1.9.3 h295c915_1 defaults markdown 3.3.4 py37h06a4308_0 defaults matplotlib 3.5.1 py37h06a4308_1 defaults matplotlib-base 3.5.1 py37ha18d171_1 defaults mkl 2021.4.0 h06a4308_640 defaults mkl-service 2.4.0 py37h7f8727e_0 defaults mkl_fft 1.3.1 py37hd3c417c_0 defaults mkl_random 1.2.2 py37h51133e4_0 defaults multidict 5.2.0 py37h7f8727e_2 defaults munkres 1.1.4 py_0 defaults ncurses 6.3 h5eee18b_3 defaults numpy 1.21.5 py37h6c91a56_3 defaults numpy-base 1.21.5 py37ha15fc14_3 defaults oauthlib 3.2.0 pyhd3eb1b0_0 defaults opencv 3.4.2 py37h6fd60c2_1 defaults openssl 1.1.1p h5eee18b_0 defaults packaging 21.3 pyhd3eb1b0_0 defaults pcre 8.45 h295c915_0 defaults pillow 9.0.1 py37h22f2fdc_0 defaults pip 21.2.2 py37h06a4308_0 defaults pixman 0.40.0 h7f8727e_1 defaults protobuf 3.20.1 py37h295c915_0 defaults py-opencv 3.4.2 py37hb342d67_1 defaults pyasn1 0.4.8 pyhd3eb1b0_0 defaults pyasn1-modules 0.2.8 py_0 defaults pycparser 2.21 pyhd3eb1b0_0 defaults pyjwt 2.1.0 py37h06a4308_0 defaults pyopenssl 21.0.0 pyhd3eb1b0_1 defaults pyparsing 3.0.4 pyhd3eb1b0_0 defaults pyqt 5.9.2 py37h05f1152_2 defaults pysocks 1.7.1 py37_1 defaults python 3.7.13 h12debd9_0 defaults python-dateutil 2.8.2 pyhd3eb1b0_0 defaults pytorch 1.11.0 py3.7_cuda11.3_cudnn8.2.0_0 pytorch pytorch-mutex 1.0 cuda pytorch qt 5.9.7 h5867ecd_1 defaults readline 8.1.2 h7f8727e_1 defaults requests 2.28.0 py37h06a4308_0 defaults requests-oauthlib 1.3.0 py_0 defaults rsa 4.7.2 pyhd3eb1b0_1 defaults scipy 1.7.3 py37hc147768_0 defaults setuptools 61.2.0 py37h06a4308_0 defaults sip 4.19.8 py37hf484d3e_0 defaults six 1.16.0 pyhd3eb1b0_1 defaults sqlite 3.38.5 hc218d9a_0 defaults tensorboard 2.6.0 py_1 defaults tensorboard-data-server 0.6.0 py37hca6d32c_0 defaults tensorboard-plugin-wit 1.6.0 py_0 defaults tk 8.6.12 h1ccaba5_0 defaults torchvision 0.2.2 py_3 pytorch tornado 6.1 py37h27cfd23_0 defaults typing-extensions 4.1.1 hd3eb1b0_0 defaults typing_extensions 4.1.1 pyh06a4308_0 defaults urllib3 1.26.9 py37h06a4308_0 defaults werkzeug 2.0.3 pyhd3eb1b0_0 defaults wheel 0.37.1 pyhd3eb1b0_0 defaults xz 5.2.5 h7f8727e_1 defaults yarl 1.6.3 py37h27cfd23_0 defaults zipp 3.8.0 py37h06a4308_0 defaults zlib 1.2.12 h7f8727e_2 defaults zstd 1.5.2 ha4553b6_0 defaults

ltkong218 commented 2 years ago

Thanks for your interest in our work. I think your problem may be the inappropriate experiment environment. I have verified that this code can be executed correctly with Python 3.7 and PyTorch 1.9.1. So please down grade your PyTorch version and try again.

NK-CS-ZZL commented 2 years ago

Thanks for your reply. I'll try it. And BTW, If it's convenient for you, I advise updating a 'requirements.txt' or 'environoment.yml'.