Problem about "Falling back to previous checkpoint" strategy

[bouquet-cc]

Hi, I'm using 3 computers to run your denoising demo (F16):

1. RTX2080Ti + Pytorch 0.4.0 + CUDA9.0 + Ubuntu 18.04
2. RTX2080Ti + Pytorch 0.4.1 + CUDA9.0 + Window 10
3. GTX1080Ti + Pytorch 0.4.0 + CUDA8.0 + Window 10

In all my computers, I found the "Falling back to previous checkpoint" strategy always happened and even happened twice within 3000 iterations. And I test the inpainting demo following the Readme and get fantastic results because there is no "Falling back to previous checkpoint" strategy in inpainting.

Is it a common phenomenon in denoising? I am wondering should I set different parameters for different images because I found the "Falling back to previous checkpoint" strategy happened in different iterations for different images.

[sc621]

I also met the same problem. My cuda version is 10.1 and i created the env with the environment.yml. I run the denoising.ipynb, and it often Falling back to previous checkpoint. I could not get the final result as goog as the paper said.

[lzw-h]

@cybouqet @sc621 I also encounter the same problem.

[TMAN1772]

+1

[hrhhao]

+1