search

Artoriuz / ArtCNN

Super-Resolution Convolutional Neural Networks as GLSL shaders for mpv
MIT License
109 stars 2 forks source link

Difference in brightness between main variant and denoise variants #9

Closed raziel711 closed 6 months ago

raziel711 commented 6 months ago

I'm not sure if this is related to the tint mentioned in https://github.com/Artoriuz/ArtCNN/commit/37b7c9c80d164a501c081ea1654ae9ae608f18d9, but I've noticed a slight difference in brightness between the main variant of ArtCNN and the denoise variants, both DN and DS.

For example, this is with ArtCNN_C4F16: ArtCNN_C4F16

and this is with ArtCNN_C4F16_DS: ArtCNN_C4F16_DS

The difference is fairly subtle, but if you look at the darker parts of the image, around the bottom left and right corners, the DS variant looks slightly darker, at least on my monitor.

Artoriuz commented 6 months ago

Yeah, it's the exact same issue. The problem here is that the "HR" dataset is being loaded into grayscale from RGB PNGs while the "LR" dataset is loaded from YCbCr JPEGs.

They are both just using cv2.IMREAD_GRAYSCALE which is probably not ideal either.

I removed the DN and DS variants from chroma and left the normal ones there because I thought the tint difference was more noticeable than the slight brightness difference, but I'm aware of the issue.

I can probably fix this by improving the training pipeline a little bit, so thanks for reminding me about it =).

raziel711 commented 6 months ago

On a related note, I also have a more general question about using ArtCNN for low resolution or poor quality sources.

Is there an anti-aliasing dataset or anti-aliasing version of Manga109 that can be used to remove more aliasing in the source itself? I would assume that training a variant to be more aggressive in removing aliasing would be naturally softer and would avoid sharpening compression artifacts as opposed to trying to remove them by denoising.

Artoriuz commented 6 months ago

@raziel711 can you try one of the two new DS variants? https://github.com/Artoriuz/ArtCNN/tree/main/YCbCr

Is there an anti-aliasing dataset or anti-aliasing version of Manga109 that can be used to remove more aliasing in the source itself? I would assume that training a variant to be more aggressive in removing aliasing would be naturally softer and would avoid sharpening compression artifacts as opposed to trying to remove them by denoising.

As far as I know we don't have any dataset specifically made to train AA models. I could easily be wrong though because I've never really went out of my way to look out for them. I do however have an old attempt at making an AA variant which I didn't really like enough to include in the repo: ArtCNN_C4F16_AA.txt

If this seems to help on your sources I could train a bigger version of it.

raziel711 commented 6 months ago

@raziel711 can you try one of the two new DS variants? https://github.com/Artoriuz/ArtCNN/tree/main/YCbCr

I've done a few comparisons, and it seems that the issue has been fixed. There isn't any difference in brightness between the standard and DS/DN YCbCr variants with the sources I've looked at.

As far as I know we don't have any dataset specifically made to train AA models. I could easily be wrong though because I've never really went out of my way to look out for them. I do however have an old attempt at making an AA variant which I didn't really like enough to include in the repo: ArtCNN_C4F16_AA.txt

If this seems to help on your sources I could train a bigger version of it.

If it's not too much trouble, I would appreciate it if you could train a larger version, but could you also include a C4F8 version as well for me to test? I know that it probably won't look nearly as good, but my GPU isn't quite strong enough to double 1080p video with the C4F16 variants.

Also, and I apologize in advance if I'm starting to seem too demanding, would it be possible to train a C4F8_AA, C4F16_AA, and C4F32_AA for the chroma layer as well, or would that require making an entirely new model?

Artoriuz commented 6 months ago

I'm retraining the luma models with the brightness fix right now, will upload them once they're done =).

About the AA experiment, I'm not quite happy about it yet and at this point I think I have way too many variants. If I changed the methodology or the dataset I'd have literally dozens of models to train which is not ideal. I think I should probably start removing them rather than adding more, but AA is a recurrent request so I'll think about it.

About AA for chroma, I don't think that should be necessary. Have you tried the DN variant? It should be relatively soft by design.

Artoriuz commented 6 months ago

Fixed by 321b70bbeb2f244e1eab92e39616dc14a3c11531