GPU Memory Usage

[moeedkundi]

I tested this demo on multiple GPUs which are Nvidia GeForce 1080 and GTX 750ti, i encountered the following issues:

1) Maximum photo size 1080 GPU could handle was 800x600, and GTX 750ti could handle max 600x480 if display is not on this gpu (750ti) and 320x240 if display is connected to this gpu. Larger sizes than this result in CUDA out of memory error as Python process takes all the available gpu memory, shown in nvidia-smi.

2) If this demo is run as a function in another program, it doesn't release GPU memory after stylization of the image.

[AliFlux]

I'm facing the same issue. The memory isn't cleared when the function ends, somehow it accumulates and crashes the program.

[midael]

Same issue here. Amazing demo/sw, but I would really love to try out this code on at least 4k UHD resolution images (3840x2160). But like others have noted, GeForce 1080 runs out of memory somewhere around 800x600 resolution.

[NVFX]

I can fix this issue.

[moeedkundi]

@NVFX do tell..

[NVFX]

It is possible to scale the image across multiple passes (or GPUs in parallel) to generate an HD style transfer without changing your code or script in any significant way. I don't have an official role on this project, but I am very interested in this problem.

[NVFX]

@moeedkundi If you extract this code locally and run demo4k.sh in your FastPhotoStyle directory. It will execute the following command:

./NvBlit -w 3840 -l 2160 -i 700 -s ./images/style1.png -c $content_image -o $output_image -x "python demo4k.py --style_image_path @style_image --content_image_path @content_image --output_image_path @output_image"

...Which will generate a 4k version of the demo.sh content.

fps4k.zip

[wesleyw72]

@NVFX What is NvBlit?

[NVFX]

@wesleyw72 It is an Ubuntu 16.04 c++ application that converts your content image into a set of interleaved tiles. The tiles are of -i 700 size in this case, but it is the largest square content image you can send your GPU. The output is automatically fed to your neural network and merged together as the final pass. The code can be easily modified to send each GPU (locally or remote) a copy of the image for processing in parallel.

There is no python version at this time and the square tile size is a simplification.

demo4k.sh downloads the content EXCEPT the following lines are commented out to work with high resolution images:

do not resize for NvBlit

convert -resize 25% content1.png content1.png;

convert -resize 50% style1.png style1.png;

[wesleyw72]

@NVFX Awesome! Is the source code available for this tool? Mainly interested in the technique used for merging the images. Is it a simple merge or does it do something clever?

[NVFX]

@wesleyw72 NvBlit interleaves the content image in the horizontal and vertical direction for inference and performs the inverse when all tiles are processed. If you run the script you should see this in the content and results folder. The smoothing step is performed at the end.

NvBlit works with just about any image processing algorithm including video, texture synthesis, etc.

Please test the app & send me your feedback. I can certainly create a python library to manage GPU resources. I will then add this work to my schedule (if approved).

[NVFX]

A quick test this morning...

+

[Yijunmaverick]

@NVFX This is really awesome! Thanks for your interests and efforts. I try your demo4k.sh with another style example. The transfer result (3.5K*1.5K, takes 4.5 minutes) is shown below:

My principle of selecting the content and style is that before running the algorithm, I could somehow imagine what the result will be like in mind (e.g., spring to winter, day to night).

Just want to double check with you. It looks that such an interleave strategy is still at the cost of sacrificing the resolution a little bit, right? When I zoom in the result, I found that details on the mountain and grass are not as clear as those in the content.

The result:

The content:

The style:

[wesleyw72]

@NVFX This is great work. Do you know of any resources where I could learn more about the interleaving technique?

[NVFX]

@wesleyw72 @Yijunmaverick There is a final change the requires modification of the style transfer algorithm. I will discuss this information with my manager and perhaps I can get involved to fix it.

To increase resolution using NvBlit interleaving algorithm:

1. Adding noise to the ease the resolution.
2. Dithering interleaved cells based on image complexity.

Disclosing the algorithm and source is the next step in a c++ & python library.

[Yijunmaverick]

@NVFX Thanks. Look forward! I could get the big picture of how such an interleave strategy works. Interested in details.