Errors evaluating 3D CNN model (Keras)

[Voxelworld]

Hello,

I'm submitting a bug report (I searched for similar issues and did not find one).

Current behavior

I build a Keras CNN which is based on 3D operations (Conv3D, MaxPooling3D, Upsampling3D, ..) and converted it with keras2onnx. When I run the ONNX models with the WinMLRunner an exception is raised:

onecoreuap\windows\windowsai\winml\dll\learningmodel.cpp(378)\Windows.AI.MachineLearning.dll!00007FFE4EF7956B: (caller: 00007FFE4EFFB6D9) Exception(2) tid(3364) 80070057 Falscher Parameter. 000001B1CEF4217C

Is Windows ML not capable of running such models?

I tried different model modifications, which all failed:

• Created a _channelsfirst and a _channelslast model version: in/out=[1, 1, 64, 64, 64] and [1, 64, 64, 64, 1] respectively.
• Used different opsets: 7, 8
• Used different converters: onnxtools 1.3.1, keras2onnx 1.3.2, winmltools 1.3.0, WinML Dashboard 0.51.
• Replaced a 'keras.layers.Conv3DTranspose' layer, because it raised another exception:

  Binding (device = CPU, iteration = 1, inputBinding = CPU, inputDataType = Tensor, deviceCreationLocation = WinML)...[SUCCESS] [FAILED] Input X must be 4-dimensional. X: {1,128,16,16,16} Evaluating (device = CPU, iteration = 1, inputBinding = CPU, inputDataType = Tensor, deviceCreationLocation = WinML)...[FAILED]

The attached model version (channels_first [1, 1, 64, 64, 64], keras2onnx 1.3.2, onnx 1.3.0, opset_version 7) runs without errors using the onnxruntime 0.3.0 python package. unet3d.channels_first.keras2onnx.onnx in models.zip

Minimal reproduction of the problem with instructions

Run attached ONNX model in WinMLRunner, or convert attached Keras model (.h5) to an .onnx model first.

Environment

Windows Build Number: Windows 10 Version 1809 (Build 17763.437)

App min and target version: WinMLRunner.exe Version: 1.0.1.0.190321.abf02d3

WinMLTools specific:

• Keras framework (keras 2.2.4, tensorflow 1.12.0)
  • WinMLTools version 1.0.1.0

Best regards, Volker

[zhangxiang1993]

Hi,

Thank you for reporting this bug.

1. After Investigation, we found that WinML only has implementation for Upsample2D, we have already filed a bug for this.
2. In order to work around it, you could write a custom operator for Upsample3D since the model runs well in OnnxRuntime.

By the way. Could you tell us more about the 3D scenario? e.g. What is the input of your model and what is used for?

[Voxelworld]

Hi Xiang, many thanks for investigating the issue and thanks for the tip!

• I didn't noticed that it's possible to implement custom operators in WinML. Is there a tutorial or code sample?

• The context of the network is a simple CNN feature detector for 3D medical data of a CT scan, similar to the one presented in kaggles Data Science Bowl 2017.

• Is there a list of supported ONNX operations? There seems to be a similar issue with the 3D version of ConvTranspose which is shown correctly in Netron: When running this operator even the ONNX runtime fails (with a meaningful error):

  FAILED 'Method run failed due to [ONNXRuntimeError] 2 INVALID_ARGUMENT Input X must be 4-dimensional. X {1,128,16,16,16}'

Best regards Volker

[zhangxiang1993]

Hi Volker,

1. For creating custom operators: you could refer the APIs as listed here and we also have a sample project for custom RELU operator .

2. ConvTranspose is another operator that we don't support 3D. And we have several other operators don't support 3D case either.

3. In Windows 10 Version 1809, we support operators of ONNX 1.2(opset 7), in Windows 10 Version 1903, we support operators of ONNX 1.3(opset8).

Cheers Xiang

[Voxelworld]

Hi Xiang, thanks for the explanation.

It's a little bit confusing since the ONNX standard describes some of the operations for the generic N-dimensional case, e.g. ConvTranspose. So the operation is documented as part of opset version 1 (ConvTranspose-1). The generic Upsample operation seems to be part of opset 7. As far as the implementation is concerned, it's not as easy, I guess. :-)

1. Can you make release notes available, to clarify which 3D ops are implemented in the respective Windows ML release?

2. You said:

   1. After Investigation, we found that WinML only has implementation for Upsample2D, we have already filed a bug for this.

   Will Upsample3D (or perhaps ConvTranspose3D) be available in the next release?

It would be great if 3D operations would be fully supported in the near future to support things like volume- and video-processing.

Cheers Volker

[Voxelworld]

Hi Xiang, any news about 3D WinML improvements in the upcoming Windows release?

Even if the example of the custom ReLu-operator looks simple, a CPU implementation of 3D ConvTranspose or Upsample would be a little painful, and I assume that it would slow down the inference very much since the 3D data (5D tensors) have to pass the CPU-GPU boundary for every node.

Cheers Volker

[zhangxiang1993]

Hi Volker,

Sorry for the super late reply, the upcoming WinML release will support your model with 3D data on CPU, but not on GPU yet.

Thanks Xiang

[Voxelworld]

Hi Xiang,

that's great news. I'm looking forward to the upcoming release.

Best regards Volker

[martinb35]

closing per @zhangxiang1993 last comment.