search

getnamo / TensorFlow-Unreal

TensorFlow plugin for the Unreal Engine.
Other
1.15k stars 211 forks source link

4.22 - Failed to load the native TensorFlow runtime. #51

Open Tokusei opened 4 years ago

Tokusei commented 4 years ago

Ok so, to confirm the machine can build TF. I installed it via Anaconda Windows 10 Python 3.6.9 install tensorflow-gpu=1.12.0 will grab CUDA9.0/cuDNN AND CUDA9.1 as part of the prerequisite libraries. This works. Builds and trains models in VS.

Using the plugin, It grabs TF1.12.0 and the associated dependencies, But it will not run using the same CUDA/cuDNN configuration (I have tried9.0 with various versions of cuDNN) It gives a missing library error. I have the path variable pointing to cuDNN as explained on the installation guide.

full log:

Traceback (most recent call last): File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module> from tensorflow.python.pywrap_tensorflow_internal import * File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in <module> _pywrap_tensorflow_internal = swig_import_helper() File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "imp.py", line 243, in load_module File "imp.py", line 343, in load_dynamic ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed. Failed to load the native TensorFlow runtime. See https://www.tensorflow.org/install/errors for some common reasons and solutions. Include the entire stack trace above this error message when asking for help. Traceback (most recent call last): File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module> from tensorflow.python.pywrap_tensorflow_internal import * File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in <module> _pywrap_tensorflow_internal = swig_import_helper() File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "imp.py", line 243, in load_module File "imp.py", line 343, in load_dynamic ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\tensorflow-ue4\Content\Scripts\TensorFlowComponent.py", line 2, in <module> import tensorflow as tf File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\__init__.py", line 24, in <module> from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\__init__.py", line 49, in <module> from tensorflow.python import pywrap_tensorflow File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 74, in <module> raise ImportError(msg) ImportError: Traceback (most recent call last): File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module> from tensorflow.python.pywrap_tensorflow_internal import * File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in <module> _pywrap_tensorflow_internal = swig_import_helper() File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "imp.py", line 243, in load_module File "imp.py", line 343, in load_dynamic ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed. Failed to load the native TensorFlow runtime. See https://www.tensorflow.org/install/errors for some common reasons and solutions. Include the entire stack trace above this error message when asking for help.

getnamo commented 4 years ago

Have you tried the cpu build first and can confirm it works? This would eliminate one potential problem area.

Couple of thoughts:

Tokusei commented 4 years ago

Hi, Yep, I also tried the CPU build and it does not work. It gives the same error. I understand it has an embedded python setup, I was using an external one to make sure the machine was capable of running 1.12.0.

Seems like it is failing to find the CUDA/cuDNN installation on the machine (The Anaconda install downloads and refs local nvidia binaries for tensorflow, and does not modify the PATH variables. (It downloads 9.0,9.1 and cuDNN7.6))

Explicitly which versions of CUDA and cuDNN should be used? As 9.0 has 4 patches, and cuDNN has multiple versions too. I have installed a new OS without Python and will try the CPU version again.

Tokusei commented 4 years ago

did the following on a fresh install (W10 :18362.418)

Output log has the following warnings:

ERROR:

LogPython: C:\Users\Administrator\Desktop\TF\Plugins\UnrealEnginePython\Binaries\Win64>echo Getting pip... Getting pip... C:\Users\Administrator\Desktop\TF\Plugins\UnrealEnginePython\Binaries\Win64>python get-pip.py Collecting pip Downloading https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl (1.4MB) Collecting setuptools Downloading https://files.pythonhosted.org/packages/6a/9a/50fadfd53ec909e4399b67c74cc7f4e883488035cfcdb90b685758fa8b34/setuptools-41.4.0-py2.py3-none-any.whl (580kB) Collecting wheel Downloading https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl Installing collected packages: pip, setuptools, wheel The script wheel.exe is installed in 'C:\Users\Administrator\Desktop\TF\Plugins\UnrealEnginePython\Binaries\Win64\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Successfully installed pip-19.3.1 setuptools-41.4.0 wheel-0.33.6 C:\Users\Administrator\Desktop\TF\Plugins\UnrealEnginePython\Binaries\Win64>Scripts\pip install -U -I pip Collecting pip Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl ERROR: To modify pip, please run the following command: c:\users\administrator\desktop\tf\plugins\unrealenginepython\binaries\win64\python.exe -m pip install -U -I pip

WARNING:

LogPython: Collecting tensorflow==1.12.0 Downloading https://files.pythonhosted.org/packages/05/cd/c171d2e33c0192b04560ce864c26eba83fed888fe5cd9ded661b2702f2ae/tensorflow-1.12.0-cp36-cp36m-win_amd64.whl (45.9MB) Collecting keras-preprocessing>=1.0.5 Downloading https://files.pythonhosted.org/packages/28/6a/8c1f62c37212d9fc441a7e26736df51ce6f0e38455816445471f10da4f0a/Keras_Preprocessing-1.1.0-py2.py3-none-any.whl (41kB) Collecting tensorboard<1.13.0,>=1.12.0 Downloading https://files.pythonhosted.org/packages/07/53/8d32ce9471c18f8d99028b7cef2e5b39ea8765bd7ef250ca05b490880971/tensorboard-1.12.2-py3-none-any.whl (3.0MB) Collecting astor>=0.6.0 Downloading https://files.pythonhosted.org/packages/d1/4f/950dfae467b384fc96bc6469de25d832534f6b4441033c39f914efd13418/astor-0.8.0-py2.py3-none-any.whl Collecting numpy>=1.13.3 Downloading https://files.pythonhosted.org/packages/55/7a/f32b39164262765b069b0fe3ec5d4b47580c9c60f7bd3588b58ba8e93a4c/numpy-1.17.3-cp36-cp36m-win_amd64.whl (12.7MB) Collecting grpcio>=1.8.6 Downloading https://files.pythonhosted.org/packages/4b/75/35bb3a14f671c34ecda9d621b5f363b02011baf67c4c0c6ce6b9e9aa4ddc/grpcio-1.24.1-cp36-cp36m-win_amd64.whl (1.8MB) Collecting six>=1.10.0 Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl Collecting termcolor>=1.1.0 Downloading https://files.pythonhosted.org/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz Requirement already satisfied: wheel>=0.26 in c:\users\administrator\desktop\tf\plugins\unrealenginepython\binaries\win64\lib\site-packages (from tensorflow==1.12.0) (0.33.6) Collecting keras-applications>=1.0.6 Downloading https://files.pythonhosted.org/packages/71/e3/19762fdfc62877ae9102edf6342d71b28fbfd9dea3d2f96a882ce099b03f/Keras_Applications-1.0.8-py3-none-any.whl (50kB) Collecting gast>=0.2.0 Downloading https://files.pythonhosted.org/packages/1f/04/4e36c33f8eb5c5b6c622a1f4859352a6acca7ab387257d4b3c191d23ec1d/gast-0.3.2.tar.gz Collecting protobuf>=3.6.1 Downloading https://files.pythonhosted.org/packages/2d/73/4a14606fa26f186e23015bc974f9010e2bbf1607f372e3bd5e82d2a62f1b/protobuf-3.10.0-cp36-cp36m-win_amd64.whl (1.1MB) Collecting absl-py>=0.1.6 Downloading https://files.pythonhosted.org/packages/3b/72/e6e483e2db953c11efa44ee21c5fdb6505c4dffa447b4263ca8af6676b62/absl-py-0.8.1.tar.gz (103kB) Collecting markdown>=2.6.8 Downloading https://files.pythonhosted.org/packages/c0/4e/fd492e91abdc2d2fcb70ef453064d980688762079397f779758e055f6575/Markdown-3.1.1-py2.py3-none-any.whl (87kB) Collecting werkzeug>=0.11.10 Downloading https://files.pythonhosted.org/packages/ce/42/3aeda98f96e85fd26180534d36570e4d18108d62ae36f87694b476b83d6f/Werkzeug-0.16.0-py2.py3-none-any.whl (327kB) Collecting h5py Downloading https://files.pythonhosted.org/packages/0b/fa/bee65d2dbdbd3611702aafd128139c53c90a1285f169ba5467aab252e27a/h5py-2.10.0-cp36-cp36m-win_amd64.whl (2.4MB) Requirement already satisfied: setuptools in c:\users\administrator\desktop\tf\plugins\unrealenginepython\binaries\win64\lib\site-packages (from protobuf>=3.6.1->tensorflow==1.12.0) (41.4.0) Building wheels for collected packages: termcolor, gast, absl-py Building wheel for termcolor (setup.py): started Building wheel for termcolor (setup.py): finished with status 'done' Created wheel for termcolor: filename=termcolor-1.1.0-cp36-none-any.whl size=4835 sha256=809d27398657e9c6847ca64f7978946ac24cbbf72c3cbac34a48075b558514a0 Stored in directory: C:\Users\Administrator\AppData\Local\pip\Cache\wheels\7c\06\54\bc84598ba1daf8f970247f550b175aaaee85f68b4b0c5ab2c6 Building wheel for gast (setup.py): started Building wheel for gast (setup.py): finished with status 'done' Created wheel for gast: filename=gast-0.3.2-cp36-none-any.whl size=9685 sha256=cfdde2bb4475edec5ce5c41f714a4f79694dbac12dc595e1541d058d7ad7ef0a Stored in directory: C:\Users\Administrator\AppData\Local\pip\Cache\wheels\59\38\c6\234dc39b4f6951a0768fbc02d5b7207137a5b1d9094f0d54bf Building wheel for absl-py (setup.py): started Building wheel for absl-py (setup.py): finished with status 'done' Created wheel for absl-py: filename=absl_py-0.8.1-cp36-none-any.whl size=121171 sha256=8f65c84279ac07610f4ac9a7f977fb2cf3df60a9aa7bfc8d80bc7764df7c82a5 Stored in directory: C:\Users\Administrator\AppData\Local\pip\Cache\wheels\a7\15\a0\0a0561549ad11cdc1bc8fa1191a353efd30facf6bfb507aefc Successfully built termcolor gast absl-py Installing collected packages: numpy, six, keras-preprocessing, grpcio, protobuf, markdown, werkzeug, tensorboard, astor, termcolor, h5py, keras-applications, gast, absl-py, tensorflow WARNING: The script f2py.exe is installed in 'c:\users\administrator\desktop\tf\plugins\unrealenginepython\binaries\win64\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script markdown_py.exe is installed in 'c:\users\administrator\desktop\tf\plugins\unrealenginepython\binaries\win64\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script tensorboard.exe is installed in 'c:\users\administrator\desktop\tf\plugins\unrealenginepython\binaries\win64\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The scripts freeze_graph.exe, saved_model_cli.exe, tensorboard.exe, tflite_convert.exe, toco.exe and toco_from_protos.exe are installed in 'c:\users\administrator\desktop\tf\plugins\unrealenginepython\binaries\win64\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Successfully installed absl-py-0.8.1 astor-0.8.0 gast-0.3.2 grpcio-1.24.1 h5py-2.10.0 keras-applications-1.0.8 keras-preprocessing-1.1.0 markdown-3.1.1 numpy-1.17.3 protobuf-3.10.0 six-1.12.0 tensorboard-1.12.2 tensorflow-1.12.0 termcolor-1.1.0 werkzeug-0.16.0

Tokusei commented 4 years ago

image Here is a screengrab of the site-packages directory. Can you confirm that all the dependencies have been downloaded?

Again, When I launch the project is gives a missing library error. If I try to run import tensorflow in the UE Python console, it returns the same error. There may be some misconfiguration with embedded setup, Is it possible to modify the parameters to download a newer version of Tensorflow, Or is the API incompatible with the blueprint interface?

Tokusei commented 4 years ago

Ok, So I have narrowed down the issue a little more. I am running a i7-965x CPU which does not have AVX Support. This explains why it fails to load TF in CPU mode. However, I am able to run it on my GPU in an environment setup in conda (CUDA9.0/cuDNN7.6.0). If I set up the same environment globally, The GPU plugin still fails to load the TF library.

getnamo commented 4 years ago

Hmm interesting. I can confirm the dependencies looks the same as I have when running 1.12.

I would defer to issues like: https://github.com/tensorflow/tensorflow/issues/19584 and maybe try some custom wheels: https://github.com/metral/tensorflow-wheels. Potentially look into your system environment variables to make sure the 9.0 is set to the cudapath (for gpu version).

Also you can downgrade the tensorflow version to pre-avx for cpu (e.g. try 1.5) which can be adjusted here: https://github.com/getnamo/tensorflow-ue4/blob/master/Content/Scripts/upymodule.json

If you do have an environment working however you can disable the embedded feature by changing: https://github.com/getnamo/UnrealEnginePython/blob/master/Source/UnrealEnginePython/UnrealEnginePython.Build.cs#L13 this should lookup your global python environment instead which should run if you know it runs externally.

Tokusei commented 4 years ago

Hi, Thanks for the response. here are the results

AVX-Enabled machine will run the CPU and GPU plugin.

Non-AVX machine: CUDA9.0 patch 4 + cudnn-9.0-windows10-x64-v7

Installing TF1.5.0 CPU by modifying the config will initalise in the basic map and return the addition and subtraction functions. in the Mnist map, it gives some errors because it is calling funcs which dont exist in 1.5.0

Installing 1.12.0 cannot init CPU as expected, It also cannot initalise the GPU version, however it is possible to initalise the GPU version of 1.12.0 on a non-avx macchine