Upscaling of assets

[simonsan]

We had a discussion in chat, I want to put that in an issue for later.

Basically someone on reddit was using some algorithms to upscale AoE2 assets.

See here:

He was using this tutorial with Python.

The idea was then, to use an upscaling algorithm in the assets converter or live in-game.

@TheJJ was proposing xBRZ for in-game use, which is really nice.

Another parallel option would be to use ESRGAN/SFTGAN for upscaling inside the assets converter. Because we don't want to require CUDA, we could change a config value from device = torch.device(‘cuda’)“ -> “ device = torch.device(‘cpu’).

I would propose that we ask the user for the upscaling algorithm and the quality like 0 (copy - fastest), 1 (copy & xbrz/in-game - faster), 2 (esrgan/sftgan-cuda, fast), 3 (esrgan/sftgan-cpu, slow). For CUDA we could just test and give an infobox to manually install CUDA if wanted (option 2 greyd out) and run the converter again afterwards (conversion stopped) or go for option 3 instead. Nevertheless, we would require pytorch then, if not already required elsewhere.

[LucasCTN]

Maybe interpolation of the animations could also be integrated using pytorch, like in this proof of concept?

https://www.youtube.com/watch?v=Ra9hBD1ZGlY

[TomJansen]

Someone in the thread used another (and better, IMO) algorithm trained on the manga109 dataset. https://www.reddit.com/r/aoe2/comments/bsn7tt/playing_around_with_esrgan_on_aoe2_assets_an/erzylpk?utm_source=share&utm_medium=web2x

[sandsmark]

esrgan etc. are fairly slow, an alternative could be waifu2x or fsrcnnx which are both able to run in realtime on an "ok" GPU, so shouldn't slow down asset conversion a lot even on a less powerful system.

probably easier to integrate the glsl versions than the various c++ implementations in the converter, the only sane (e. g. sane build system) machine learning library is dlib, and noone uses that.

waifu2x has a pre-trained glsl implementation here; https://github.com/yui0/waifu2x-glsl fsrcnn has pre-trained glsl shaders here; https://github.com/igv/FSRCNN-TensorFlow/releases

an idea would be to use the code in the FSRCNN-TensorFlow repo and train it on aoe graphics upscaled with esrgan, or maybe even on the definitive edition assets. maybe even the absurdly large "enhanced graphics": https://store.steampowered.com/app/1039811/Enhanced_Graphics_Pack/

[simonsan]

Pretty good Wiki and recommended read: https://upscale.wiki/wiki/Model_Database

Image Super-Resolution Using Deep Convolutional Networks (Leaderboard with Implementations) https://paperswithcode.com/task/image-super-resolution

[simonsan]

So AoC 1.0c with opengl and shader-based upscaling with xbrz looks like this:

[sandsmark]

The ugliest there is the fog of war, but if that's implemented with normal alpha blending or something instead of blindly copying genie that shouldn't be a problem.

and fwiw, it's probably obvious, but this makes it possible to get close to the "enhanced graphics pack"-quality without people having to download 30GB of extra assets.

[simonsan]

I used that by the way to use OpenGL with AoC: https://github.com/CnCNet/cnc-ddraw

I'm currently also working on upscaling all of the assets with ESRGAN models (@sandsmark: you find that project in the SiegeEngineers discord under assets-upscaling). File based upscaling looks much nicer at least for the buildings, but it will need also a bigger amount of storage on your device that's true - saying that from the quality I personally prefer the pre-upscaled method over the in-game one. Neither waifu2x has convinced me, nor a simple manga109 dataset. I'm currently using around 18 different models and merging together the results depending which ones give the best details, the best surfaces etc. That said: There is not a one fits all model, so I'm quite in favour of manually upscaling them and putting some love inside than just letting a shader or some other stuff running randomly over it.

If we commit to it as a community, we probably can upload them as a mod to the Steam workshop and let that take care of the distribution. Some hope would be as well to have these upscaled assets as a classic mode in AoE2DE so that there would be the possibility of a Enhanced Classic Graphics Pack - with the same distribution possibility over Steam. But that stuff is something for the far future I guess.

First there need to be some work done.

Here is a collaborative Sheet which assets still need some work: https://ethercalc.org/3vrevcrhm3b3

This is a teaser for what we aim for: