search

ivanstepanovftw / artistic-style-transfer-for-videos

Artistic style transfer for video, using image frame style transfer and optical flow estimation
Apache License 2.0
0 stars 0 forks source link

Strange output with red noise point #3

Open AlexCHENSiyu opened 5 months ago

AlexCHENSiyu commented 5 months ago

styled_38_0 I can get this good output yesterday. but today when I ran it again, I get this strange output: styled_39_1

Have you ever encountered this problem before? Or you can give me some suggestions?

AlexCHENSiyu commented 5 months ago

ok, probably I find the reason. BTW , this part should be written like this: image add ['arr_0'] to it. otherwise cannot run.

ivanstepanovftw commented 5 months ago

I had this when I did not clamp styled image: https://github.com/ivanstepanovftw/artistic-style-transfer-for-videos/blob/b42b79d17fd1b72917b55100182c6f5f3d097d92/main_video_join.py#L57 Similar as it is performed in the closure: https://github.com/ivanstepanovftw/artistic-style-transfer-for-videos/blob/b42b79d17fd1b72917b55100182c6f5f3d097d92/main_video.py#L342 I actually thinking about adding loss for out of bounds pixels, so the optimizer would not be focused on these pixels.

AlexCHENSiyu commented 5 months ago

yes, right after I posted the issue, I found this mistake of mine. I read the paper carefully and I realize that long-term temporal consistency requires heavy-duty computation. It might be hard for me to achieve that.

AlexCHENSiyu commented 5 months ago

BTW, do you know any comparative standard to measure the quality of style-transferred images between different pre-train model?

AlexCHENSiyu commented 5 months ago

and any reason why you choose efficientnet-b0 as the best model

ivanstepanovftw commented 5 months ago

BTW, do you know any comparative standard to measure the quality of style-transferred images between different pre-train model?

Unfortunately, no.

and any reason why you choose efficientnet-b0 as the best model

I am not sure if it is best for style transfer. EfficientNet chosen as it is efficient, since I do not have local GPU.

Also, in the paper "CNN Filter DB: An Empirical Investigation of Trained Convolutional Filters" (arxiv:2203.15331v2), they say that "learned filters do not significantly differ across models trained for various tasks, except for extreme outliers such as GAN-Discriminators."