search

Algolzw / image-restoration-sde

Image Restoration with Mean-Reverting Stochastic Differential Equations, ICML 2023. Winning solution of the NTIRE 2023 Image Shadow Removal Challenge.
https://algolzw.github.io/ir-sde/index.html
MIT License
521 stars 39 forks source link

Repeat 100it [00:45, 2.22it/s] #51

Closed xiaoyeqiufang closed 8 months ago

xiaoyeqiufang commented 9 months ago

What does the repeated statement below mean? I tried to reproduce it, but I've been stuck at this point, and it seems like it doesn't finish running in a short period of time.

Path already exists. Rename it to [/root/image-restoration-sde-main/experiments/deraining/ir-sde_archived_231005-125919] cosine schedule 100it [00:30, 3.29it/s] 100it [00:25, 3.88it/s] 100it [00:47, 2.10it/s] 100it [00:43, 2.32it/s] 100it [00:25, 3.88it/s] 100it [00:41, 2.44it/s] 100it [00:52, 1.92it/s] 100it [00:51, 1.96it/s] 100it [00:45, 2.22it/s] 100it [00:48, 2.06it/s] 100it [00:45, 2.22it/s] 100it [00:45, 2.22it/s] 100it [00:45, 2.22it/s] 100it [00:25, 3.88it/s] 100it [00:25, 3.88it/s] 100it [00:25, 3.88it/s] 100it [00:25, 3.88it/s] 100it [00:25, 3.88it/s] 100it [00:25, 3.88it/s] 100it [00:25, 3.88it/s] 100it [00:42, 2.36it/s] 100it [00:25, 3.88it/s] 100it [00:25, 3.88it/s] 100it [00:45, 2.22it/s] 100it [00:25, 3.88it/s] 100it [00:25, 3.88it/s] 100it [00:25, 3.88it/s] 100it [00:25, 3.88it/s] 100it [00:25, 3.88it/s] 100it [00:25, 3.88it/s] 100it [00:25, 3.88it/s] 100it [00:25, 3.88it/s] 100it [00:45, 2.22it/s] 100it [00:25, 3.88it/s] 100it [00:25, 3.88it/s] 36it [00:09, 3.85it/s]

Algolzw commented 9 months ago

I haven't met this problem before. Sometimes it will be stuck due to the memory issue. Have you checked your memory and GPU usage?

xiaoyeqiufang commented 8 months ago

I haven't met this problem before. Sometimes it will be stuck due to the memory issue. Have you checked your memory and GPU usage?

This should be a validation process during the training process, with each representative validating an image.

wangwangzhang9527 commented 8 months ago

I haven't met this problem before. Sometimes it will be stuck due to the memory issue. Have you checked your memory and GPU usage?

Hi! I'm having the same problem with the image deblurring training process, this seems to be taking too long to validate, it takes about 4 hours to validate on the 4090, how long does your test or validation take? Am I setting it up wrong somewhere?

Algolzw commented 8 months ago

@wangwangzhang9527 Hi! For deblurring I highly recommend you use less number of images for validation. And you can only test once on the full test set using the last checkpoint.

wangwangzhang9527 commented 8 months ago

@wangwangzhang9527 Hi! For deblurring I highly recommend you use less number of images for validation. And you can only test once on the full test set using the last checkpoint.

Thank you for your timely and useful response. This solved my confusion.