search

murdockfpv / murdockfpv.github.io

Build a Jekyll blog in minutes, without touching the command line.
Other
0 stars 1 forks source link

Ultra High-res style transfer "Van-Gogh-ify" with highest amount of texture applied from style #76

Open juliusakula opened 2 years ago

juliusakula commented 2 years ago

The 30,000 x 23,000 version of the Van Gogh Starry Night painting, available on wikipedia, is incredible. Zoom in and zoom in further, see the thick gobs of paint in places.

!1 - The moon on Van gogh's Starry night [zoomed in]

See the bare canvas between the paint.

!2 - Bare canvas texture zoomed in

And imagine that a neural network can be trained on such details, and that training be used to stylize a photo. The end result of a style transfer using a "gigapixel" version of a style image retains amazing detail:

!3 gigapixel starry night stylized photo

Even with the most powerful graphics processors available in the world today, it takes those machines several hours to train the neural network (step 1) - and more time still to apply the knowledge of the neural network to your photograph, in order to stylize it (step 2).

!4 Zoomed in generation via gigapixel

Even though I'm generating these with artificial intelligence, a substantial amount of work is involved:

What follows is an in-depth explanation of how to do this yourself - knowledge of command line utilities would help, you will need to install many different components in order to execute this code. In the end you will have a powerful tool at your disposal.

I'm going to outline the basic process to stylize your own photo at up to 4K resolution, on your own computer (if you have a good enough computer (RTX 2060 or above))

Setp 1. Take a photo.

!5 - Starting photo

As a mental excercise, consider that a 3840 x 2160 image (A 16:9 image still from a 4K video) is twice the size of a 1920 x 1080 image (1080p), right? wrong! 4K is 4x 1080p (hd). As images expand in size in both directions, the amount of pixels grows exponentially. 2x is 4x. 3x is 16x. 4x is 64x. etc. That being said, the larger the image the better. My solution to problem of this taking weeks to run the program on my laptop - is renting the best graphics cards in the world by the hour, and running the program, saving my result, and closing my cloud computing instance right after or automatically. Quite an involved process, but the results are worth it:

Your output by default in this program is the same size as your input. If you want this to take under 30 minutes (Your computer will likely freeze up the entire time the neural network trains (when the program runs)) -- Resize your input image to something small like 384 x 216 to make a thumbnail sized output. For a large print, you do need as high resolution as possible and may not be possible on your computer's hardware.

2 Download a version of Starry Night from wikimedia commons

(They have a 200MB gigapixel image which would yield the best results, if your computer can handle it. Try the 2560 x 2027 version to start. This is the image all the style is extracted from. Really you could use any image, painting or art as the style image - but we are "Van-Gogh-ifying" the image so we use starry night.

3 Download all required drivers, libraries and other dependencies in order to run the docker-neural-styles open source software

juliusakula commented 2 years ago

4 Enter the WSL command line and navigate to the neural-style-docker directory#63

cd ../..
cd Users/justin/git/neural-style-docker

5 move the starry night image and your photo into the neural-style-docker/tests/contents directory

6 Execute the program and pass in your "Style Image" and "Target Image" as inputs, like so:

docker run --rm -v /mnt/c/Users/justi/git/neural-style-docker/tests/contents:/images --gpus all albarji/neural-style --content target_photo.jpg --style starry_night.jpg
juliusakula commented 2 years ago

Allow the program to run as long as necessary. Make note of when you start the program. When you have your output, check the time it was created. That's how you'll know it took X amount of time to create that image at that resolution, and can make a decision about trying a larger or smaller resolution the next time you run this code.