Color loss while upscaling videos

[JFMugen]

From what I can see (when I compare images from temp folders, there is no color difference) it is because of FFmpeg, but trying to make settings as lossless as possible didn't help :/ For example, left is original while right is upscaled. It is most obvious at red-purple colors.

[Moebits]

Since this is a ffmpeg issue you have to open up the bug with them and i'll update the ffmpeg version whenever they fix it. I can see that some bugs still haven't been fixed after 10 years, so it might take awhile... https://trac.ffmpeg.org

Regardless this is very easy to fix, open the video in a video editor and bump up the saturation slightly then re-export. Since this isn't an issue with my code there is nothing for me to fix.

[JFMugen]

Since this is a ffmpeg issue you have to open up the bug with them and i'll update the ffmpeg version whenever they fix it. I can see that some bugs still haven't been fixed after 10 years, so it might take awhile... https://trac.ffmpeg.org

Regardless this is very easy to fix, open the video in a video editor and bump up the saturation slightly then re-export. Since this isn't an issue with my code there is nothing for me to fix.

I am not sure, but maybe this can be solved with the command you use as well? Look here

https://superuser.com/questions/1511254/ffmpeg-video-compression-desaturating-colors-losing-color-information

[Moebits]

Ok, can you post the video that you posted (trim it if its long) if that answer is correct than adding this to the command should fix it. -color_primaries bt2020 -color_trc bt709 -colorspace bt2020_ncl -color_range pc -bsf:v h264_metadata=video_full_range_flag=1:colour_primaries=9:transfer_characteristics=1:matrix_coefficients=9

I'm not sure if this will break videos that are not in the bt.2020 color space, so if this issue only occurs on certain videos I would instead advise you to run that command yourself in order to fix those vidoes.

[JFMugen]

Ok, can you post the video that you posted (trim it if its long)

https://user-images.githubusercontent.com/61559703/182943886-42899462-3c0a-4cfe-9216-ec8d58246239.mp4

I am not sure if the answer is right or wrong but makes kinda sense. Because some colors are more saturated while others less. I would try it myself but I am pretty noob about how to use ffmpeg :/

[Moebits]

Ok, I tried adding those flags but it makes the colors TOO saturated.

I also tried -color_primaries 5 -colorspace 5 -color_trc 6 instead and while the colors aren't exactly the same they are a closer match, so I think I'll stick with this one.

[JFMugen]

Well, that is definitely annoying. May I ask which code you use in the end (all the code) so I can ask in ffmpeg forums?

[Moebits]

The code is in a different repo, here: https://github.com/Tenpi/waifu2x/blob/2cb76f0e2b296caedfbed5621a485a9f6bfef414/waifu2x.ts#L624

The full command should look something like ffmpeg -i frame%d.png -r 24 -vcodec libx264 -pix_fmt yuv420p -movflags +faststart -crf 16 -color_primaries 5 -colorspace 5 -color_trc 6 -vf crop=trunc(iw/2)*2:trunc(ih/2)*2 output.mp4

Good luck!

[JFMugen]

@Tenpi I think I found the answer with some help ^^

https://old.reddit.com/r/ffmpeg/comments/whq5sm/color_losschange_with_ffmpeg/

Please read the post, he made some awesome explanations.

[Moebits]

Ok, you will need to edit the videos manually if they have wrong colors since there is no way for me to detect it automatically but I made the default color space bt709. I will leave this issue open if others experience the same problem.

[JFMugen]

Ok, you will need to edit the videos manually if they have wrong colors since there is no way for me to detect it automatically but I made the default color space bt709. I will leave this issue open if others experience the same problem.

Well, weirdly enough it worked better for me (several videos) when I used NTSC DVD (-color_primaries smpte170m -colorspace smpte170m -color_trc smpte170m) instead of bt709. I just installed ffmpeg for the first time just for this haha :D

I am not %100 sure if you should change it again or not (its up to you), but I will ask one last thing about this. Can you add an option to remove upscaled images to recycle bin? So we can restore if colors are strange and do our own thing.

[Moebits]

I am not %100 sure if you should change it again or not (its up to you), but I will ask one last thing about this. Can you add an option to remove upscaled images to recycle bin? So we can restore if colors are strange and do our own thing.

You can fix this using the original video without any re-encoding, all it does is change the metadata: ffmpeg -i input.mp4 -c copy -color_primaries smpte170m -colorspace smpte170m -color_trc smpte170m output.mp4

You seem to be upscaling old anime that is in SD which is why that color space works better for you. The bt709 is mostly used in HD videos. I added a checkbox in the settings to toggle between bt709 and smpte170m.

[JFMugen]

I didn't know that, thanks ^^

Thanks for the toggle as well. Just a thought tho, isn't it expected to use SD videos since this is an upscaler :D

[Moebits]

People can still upscale HD videos if they want to make them 4K or higher. But I think upscaling SD videos will be more common so I made that setting on by default.

[JFMugen]

Hopefully this will fix most color-related problems. Thanks for the effort, it was more complicated than I expected :)

[JFMugen]

Well, sorry to reopen this but looks like there is a problem. When you don't select "SD Colorspace", output files don't have any audio.

[Moebits]

Please put separate bugs into separate issues, at this time I don't know why that could be happening

[JFMugen]

Please put separate bugs into separate issues, at this time I don't know why that could be happening

I thought the issue is related to the recent "SD Colorspace" tick box you added. Mb

[Moebits]

I'll consider this solved the SD Colorspace should cover most cases