search

jantic / DeOldify

A Deep Learning based project for colorizing and restoring old images (and video!)
MIT License
18.05k stars 2.58k forks source link

Add colorization hinting #32

Closed t-mullen closed 5 years ago

t-mullen commented 6 years ago

Would be useful to manually correct some of the colorization glitches, or make the output more historically accurate.

Ideally, a secondary image with user-defined colour hints could be provided to both the generator and critic. Generating training hints that look like user brush-strokes might be a bit difficult, although I'm not sure how much that would matter.

Another, easier option would be to specify only a palette: http://openaccess.thecvf.com/content_cvpr_2017_workshops/w12/papers/Cho_PaletteNet_Image_Recolorization_CVPR_2017_paper.pdf

Awesome project by the way, can't wait to see how it develops!

jantic commented 6 years ago

Phew....yeah I've had this thought spring into mind. I'll chew on it for a while because it's not immediately obvious to me how to get this to work well with what I currently have. But if it did, it'd be a really sweet combination!

MayeulC commented 6 years ago

On a related note, it might be interesting to have this for #18, in order to keep colors consistent from one frame to another. The previous frame could be compared with the next, and if reasonably similar (might be worth a look at what video codecs are doing), provide colour from the previous frame as a seed/hint value for the next frame. I mentioned modern video codecs as they should be able to relatively efficiently map objects displacement from one frame to the other, so this could be used to move hints to the right place, from one frame to another. But as a first approach, hinting could be used only for static parts of the images.

And of course, this might not be needed at all! (though I doubt it)

jantic commented 5 years ago

This is a nice to have in the way beyond future, to be honest. Putting in a todo list we're maintaining separately.