search

nagadomi / waifu2x

Image Super-Resolution for Anime-Style Art
http://waifu2x.udp.jp/
MIT License
27.44k stars 2.71k forks source link

Waifu2x in real time #117

Open Mickevincent opened 8 years ago

Mickevincent commented 8 years ago

Hello! I'am guessing nagadomi is the creator of Waifu2x? Anyway, I have something I want to discuss. Waifu2x is the best tool I have ever seen to upscale old PS1 pre-rendered backgrounds. I don't know if you have played these games, I don't even know if you are a gamer. But remember that the originals were rendered in somewhere around 320x240 pixels. That is such a low resolution even phones ten year back put it to shame. In this batch you can see some examples of backgrounds upscaled and filtered with Waifu2x and highest noise reduction.

http://imgur.com/a/p4hxg

No filter for any emulator does it as good as this. So I'am extremely interested in getting this ported as a shader or such for PS1 emulation.

Now I have asked around at different githubs and have got very different answers from people who do this kind of work. It seems that it actualy could be possible, however it would be alot of hard work. One issue is how demanding the Waifu2x is. I'am gonna pretend that you are not a gamer now just in case. I have a gtx 980Ti wich is at the moment one of the strongest gpu's in the world. It takes about 0.2 - 0.6 seconds for it to do one picture. (Waifu cafe) But then you gotta remember that we talk about PC-hardware wich always evolves fast. Sometime this year Nvidia will realese even stronger GPU's. So I'am wondering how well optimized is Waifu2x? Can it be rewritten to do it faster? I have also opened a Kickstarter for this. It is not yet approved. But if it gets succesfull there will be credits to get for those involved in this. I'am also wondering if it would be ok with you to do this? It is after all your code (I think).

I'am looking forward for an answer! :) /Mike

nagadomi commented 8 years ago

Currently, I am developing a new model that is 2x faster than current model and better upscaling quality. (on GTX760 and 320x240 image, current model 0.77 sec, new model 0.41 sec) But it is not reached realtime (30FPS=0.033sec, 60FPS=0.0166sec). If you train a small(shallow) CNN model, It will be faster(quality down a little) but it cannot reach reailtime, I think.

Mickevincent commented 8 years ago

So with the new model, if it would take 0.41 with one certain picture on a 760, it could theoreticly be done in twice the time, about 0.20 sec on a 980Ti. Pherhaps even faster. Also there has been talk about a gtx 1080Ti coming this year, wich will give even more gpu power. Besides most PS1 games run in around 20fps :) So how would you feel if your code were used for such a project? This is good news :)

nagadomi commented 8 years ago

waifu2x is licensed under the MIT License. You can free to use my code under the MIT License.

Mickevincent commented 8 years ago

Ok, thank you very much. If this ever happens, I will tell you :) One more thing, when do you think the new version will be finnished?

nagadomi commented 8 years ago

new version is available at https://github.com/nagadomi/waifu2x/tree/upconv (models/upconv_7/art). Now this model has problem in rare case(small pixel art). I will release the fixed versin in a few months.

Mickevincent commented 8 years ago

Ok, again, thanx!

Naville commented 5 years ago

Sorry for necromancing. But is this now possible with RTX cards’ tensor cores on a 2080Ti?

quartzsaber commented 5 years ago

Sorry for necromancing. But is this now possible with RTX cards’ tensor cores on a 2080Ti?

I don't believe so.

First, according to a comment on tensorflow issue tracker, their computation power is severely limited.

Second, waifu2x's models aren't that deep. A shallow network cannot utilize all computational power of GPU because it is specialized in doing infrequent large calculations, not frequent small calculations.

I might be wrong, but it definetely won't be fast enough for real time uses.

Naville commented 5 years ago

@min050820 I'm still a bit confused though as someone else stated it's not limited.

Oh well I guess inb4 I'm able to waste 6500GBP on an RTX 8000, one can only keep dreaming

EDIT: Thanks for the info

quartzsaber commented 5 years ago

@Naville Sorry for the confusion. By 'limited', I meant it not being powerful enough. Maybe if waifu2x gets more shallow (=lightweight) model, it could be possible to run in real-time, just like ESPCN, which claims to be able to run in real time with Titan X.