CNN in Unity gives wrong output after inputting 3D Array

Unity-Technologies / ml-agents

The Unity Machine Learning Agents Toolkit (ML-Agents) is an open-source project that enables games and simulations to serve as environments for training intelligent agents using deep reinforcement learning and imitation learning.

https://unity.com/products/machine-learning-agents

Other

17.08k stars 4.15k forks source link

CNN in Unity gives wrong output after inputting 3D Array #335

Closed lagidigu closed 6 years ago

lagidigu commented 6 years ago

Hi,

My CNN takes in a Float32 Tensor of shape (128, 6, 500) in order to output a long[] of size 128.

In C#, I enter the input in the form of a 3D array (float[128, 6, 500]). Everything else is followed to the teeth, but the output given is wrong, and I get no error messages. Is multidimensional input even supported?

If no, would there be a way for me to implement it?

Cheers and thanks a lot in advance.

lagidigu commented 6 years ago

Can anyone help me out? Still haven't been able to find a fix... :/

awjuliani commented 6 years ago

Hi @lagidigu,

Apologies for the delayed response. Can you describe in what way your output seems to be incorrect? I assume you are using a pre-trained network that you've imported from outside of Unity? There should be no theoretical reason why this would not be working for you, unless there is some mismatch between supported operators in your original TF version and the version we supply (1.2). It may also be helpful to read through some of the documentation for TF#, or ask in their GitHub issues, as the core developers of the bindings are active there: https://github.com/migueldeicaza/TensorFlowSharp.

lagidigu commented 6 years ago

Hi @awjuliani,

Thanks for the answer! My network is a CNN that receives EEG data, and either outputs 0 or 1, for the corresponding class. When I feed samples from a specific dataset into it, the classification should be 0 more often than 1, but it is the opposite, which makes no sense. The samples from the dataset are the same as the ones that I used to train the network. I tested the frozen graph in python by loading it again, and I obtain a classification that reflects the high accuracy of the initial trained net. In Unity, it seems to be completely random.

I also posted a thread in the TensorFlowSharp forums.

vincentpierre commented 6 years ago

Can you try with a specific image for which the results differ? Test the image with python and then with TensorflowSharp? This would help you debug the network by outputting intermediate layers' values.
The problem could come from going from image to float array. Make sure the C# process and the Python process give you the same float array.
If the network does not crash in Unity it means it ran the TensorFlow library of Unity. Like Arthur pointed out, it could be a discrepancy of versions of TensorFlow. Maybe downgrade your python version

eshvk commented 6 years ago

Hi @lagidigu, Thanks for reaching out to us. Hopefully you were able to resolve your issue. We are closing this due to inactivity, but if you need additional assistance, feel free to reopen the issue.

lock[bot] commented 4 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.