Evaluation of CNTK nodes using custom data types

[e-thereal]

I'm using CNTK to train a 3D convolutional neural networks, which takes 3D images as input and also produces 3D images as output and I've got two questions regarding the evaluation of such a model.

I've written a data deserializer based on the image deserializer class in order to input 3D images in our own data format into CNTK for training. To evaluate the model, I'm given two options: using CNTK.exe or using the C++ evaluation interface, but both methods have they own limitations. The C++ evaluation interface is very flexible in terms of handling inputs and outputs of the model, and it allows the evaluation on 3D images. However, it currently does not support the evaluation using CUDA, so the evaluation of a model is rather slow.

1. Are there any plans to support GPU evaluation using the C++ evaluation interface?

Alternatively, I could use CNTK.exe directly. However, our network produces 3D images as output and I was wondering:

2. Is there a way to also write a custom data serializer?

I've seen that this seemed to be possible using the old Reader/Writer theme, so I was wondering if this is also possible using the new deserializer method.

Thank you so much.

[frankseide]

The Wiki https://github.com/Microsoft/CNTK/wiki/CNTK-Evaluation-Overview says that you can use the GPU. Did you try and it did not work, or did you see some other place that said you couldn't?

Looking at the source code, EvalDLL uses the same function to load the model (which selects the CPU/GPU) as any other part of CNTK. So you should be able to say “deviceId=0” in your configuration string. The .Net wrapper has a method CreateNetwork() that takes a device id.

As for serialization, this will come, but does not exist yet.

Within a few weeks we will have the ability to run CNTK functionality from Python programs through an API. Then an alternative could be to write data with a Python script.

[e-thereal]

@frankseide Thanks for your reply. I haven't tried using EvalDLL in GPU mode and it seemed odd, that EvalDLL would use a different evaluation engine in the first place.

On your referenced Wiki page, in section "Evaluating a model programmatically in Windows" subsection "Using the C++ library", it says:

The EvalDLL.dll is a Windows dll that enables programmatic model evaluation. The dll can only be used in CPU only mode (no GPU is used).

I will try if I can use the GPU and report back my findings.

[zhouwangzw]

To clarify: you can leverage GPU if you evaluate the model using cntk.exe. However, the current EvalDLL is built with CPU only flavor, so it would not use GPU capability. And the Nuget package for managed Eval contains also CPU only bits. Please refer to the Wiki https://github.com/Microsoft/CNTK/wiki/CNTK-Evaluation-Overview for details.

[e-thereal]

@zhouwangzw Does that mean that I can use the GPU in EvalDLL, when I've compiled CNTK myself with GPU support? I got side tracked so I still haven't had the time to actually test it.

[zhouwangzw]

@e-thereal Sure. You can try to build EvalDll with GPU and test whether it leverages GPU capability. I have built the EvalDll with GPU under Linux, but have not deeply checked whether it uses the GPU capability or not.

[e-thereal]

@frankseide @zhouwangzw I've just tried the C++ integration under Linux with our custom CNTK build that includes GPU support and I can confirm that it is possible to use the GPU by setting the deviceId in the config string, as suggested by @frankseide.

Thanks for all your feedback. Would be great to have that in the documentation. E.g., just state that the prebuild EvalDll that ships with the binary package is CPU only (if that is the case), but with a custom built EvalDll, it is possible to use the GPU.

[zhouwangzw]

@e-thereal Thanks for your effort for verification. It is good to know that the Eval really uses GPU if it is built with GPU flavor. We will update Wiki too. Just curious, do you see any performance benefit comparing CPU-only Eval?

Thanks, Zhou

[e-thereal]

@zhouwangzw Our networks benefit a lot from using the GPU. While it takes about 10 to 20 minutes to process one image through the network on the CPU, the same can be done in 6 seconds using a K20.

[zhouwangzw]

@e-thereal Cool! Thank you!