Blocking/Sketching artifacts after sudden convergence

[Anguliachao]

Dear author, thank you for the detailed script as tutorial, I'm also trying to train a tile version controlnet as learning purpose, while I find it difficult to replicate the official tile-Controlnet results, I manged to find your repo quite useful and try to replicate as my first step. However I found your max training steps are just max capped at 10K steps and I set more max training steps . I observed sudden convergence after 3-5K steps by using 4xV100 gpu, however while the training loss remains stable, I observed the validation outputs have certain visual artifacts appeared , such as blocking, downsampling artifacts, unexpected sketches or noise would occur after training for longer than 20k steps (below pictures attached, step 4k vs step30k), so I was confused whether this occur to you or have u ever trained a tile model with longer steps(>10K)/multi-scale fine-tuning, would you care to share some tips or kind advice? Many thanks

[zjysteven]

Hi. No unfortunately I have never trained for that many steps, so I'm afraid that I can't help much on that.

[Anguliachao]

Noted, thank you for the prompt response.