Open jarrellmark opened 7 years ago
Hi @jarrellmark
Thanks for reporting this!
That has been addressed in 5cc8cdd58f3324c81eeab3e9a0af47754716e7fc. Could you please try it out?
SYCL default allocator did not take alignment into consideration. That now has been addressed in Eigen, where we are passing the required alignment to the custom allocator. C++ is great!
Thanks,
Hey @lukeiwanski,
The IsAligned() message went away, but I'm getting this message now:
2017-03-20 21:30:14.765017: W ./tensorflow/core/common_runtime/sycl/sycl_util.h:44] No OpenCL GPU found that is supported by ComputeCpp, trying OpenCL CPU
Is there a way to force the GPU?
Currently we have an issue with memory alignment on Intel GPUs and have set the Intel GPU as "blacklisted" in Eigen. This means Eigen will not try to target Intel GPUs at the moment. We are working on a resolution for this and will update you when we have a fix available.
Thanks, Luke.
I appreciate it and am excited about the progress that tensorflow-opencl is making.
Hi @lukeiwanski,
I'm having the same issue and was wondering if you have added Eigen support for Intel GPUs yet. If not, is there some way I can un-blacklist the Intel GPU?
Thanks for your hard work on this project!
Can you give it a spin on this branch: https://github.com/lukeiwanski/tensorflow/tree/dev/eigen_mehdi ?
That fixed the error, thanks a lot!
An unrelated question, tensorflow keeps telling me I'm running on a SYCL device, but then it calls that device a CPU. When I run sess = tf.Session(config=tf.ConfigProto(log_device_placement=True)), I get the following output:
/job:localhost/replica:0/task:0/device:SYCL:0 -> id: 0, type: CPU, name: Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz, vendor: Intel(R) Corporation, profile: FULL_PROFILE
Running tensorflow.python.client.device_lib.list_local_devices() gives me the following:
[name: "/cpu:0" device_type: "CPU" memory_limit: 268435456 locality { } incarnation: 177593382533810523, name: "/device:SYCL:0" device_type: "SYCL" memory_limit: 268435456 locality { } incarnation: 1258559034356206920 physical_device_desc: "id: 0, type: CPU, name: Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz, vendor: Intel(R) Corporation, profile: FULL_PROFILE"]
However, this device is NOT my GPU, as can be seen from when I run clinfo:
Platform Name Intel(R) OpenCL Number of devices 2 Device Name Intel(R) HD Graphics Device Vendor Intel(R) Corporation Device Vendor ID 0x8086 Device Version OpenCL 2.0 Driver Version r5.0.63503 Device OpenCL C Version OpenCL C 2.0 Device Type GPU Device Profile FULL_PROFILE
. . .
Device Name Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz Device Vendor Intel(R) Corporation Device Vendor ID 0x8086 Device Version OpenCL 2.0 (Build 475) Driver Version 1.2.0.475 Device OpenCL C Version OpenCL C 2.0 Device Type CPU Device Profile FULL_PROFILE
Thanks for all your help already!
However, I am later getting this error when I try to run a simple keras model (just 2 dense layers): InternalError: Unknown error detected on device /job:localhost/replica:0/task:0/device:SYCL:0
That's interesting.. could you provide code to reproduce that issue?
I'm having trouble reproducing this issue because the code seems to just be hanging (I'm getting a lot of these messages: ./tensorflow/core/common_runtime/executor.cc:1556] Process node: 48 step 2 mul_3 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:SYCL:0"](beta_1/read, Variable/read) is dead: 0
But here's my code: from keras.models import Sequential from keras.layers import Dense from keras import optimizers model = Sequential() model.add(Dense(32, input_shape=(timesteps, D_in))) model.add(Dense(D_out)) model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) model.fit(X_train, y_train, batch_size=N, epochs=5, validation_data=(X_test, y_test))
Ah, ok I've reproduced the earlier error by using LSTM layers. It may be unreasonable for me to expect LSTM layers to work, but I am also having trouble with just dense layers (see above). Here's my code:
from keras.models import Sequential from keras.layers import LSTM, Dense from keras.layers.wrappers import TimeDistributed from keras import optimizers model = Sequential() model.add(LSTM(32, return_sequences=True, input_dim=D_in, input_length=timesteps)) model.add(LSTM(32, return_sequences=True)) model.add(TimeDistributed(Dense(D_out, activation='softmax'))) model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) model.fit(X_train, y_train, batch_size=N, epochs=1, validation_data=(X_test, y_test))
And here's a trace of the error message: InternalError Traceback (most recent call last)
Hi @lukeiwanski
I am having the same issue (Check failed: IsAligned()
) with tf-coriander
(https://github.com/hughperkins/tf-coriander) using a Mali T-728 GPU.
Do you have a patch I could try to fix this? Or an advice on how to go about fixing it? Thanks!
Ping on this issue
Thanks for the great work on tensorflow-opencl. It's really great.
Summary
I'm getting a runtime error for almost all tensorflow programs:
2017-03-18 12:52:52.241954: F ./tensorflow/core/framework/tensor.cc:488] Check failed: IsAligned()
Aborted (core dumped)
Environment Description
I have an Intel HD Graphics 5500 GPU with the Intel Broadwell i5 CPU x64. I'm using Intel's OpenCL drivers from here: https://software.intel.com/en-us/articles/opencl-drivers .
The OS is Ubuntu 16.04 LTS.
Python version is 3.5. It's running in a conda environment using Anaconda's versions of python, numpy, scipy, pyyaml, h5py, pandas, and jupyter.
However, the tensorflow pip package was compiled using Ubuntu's version of everything as per the compile from source instructions. I disabled Anaconda by removing it from ~/.bashrc, compiled the pip package, re-enabled Anaconda, activated the conda environment, and installed the pip package into the conda environment.
Steps to Reproduce
Here's the only tensorflow program I tried that did not fail:
Changing
NUM_ROWS
andNUM_COLUMNS
to even 1200 resulted in the error above.I also installed keras into the same conda environment using
pip install keras
and ran this script: https://github.com/fchollet/keras/blob/master/examples/lstm_text_generation.py . This resulted in the same error:Check failed: IsAligned()
. The error is displayed afterBuild model...
is outputted to the console.Commit Hash (
git rev-parse HEAD
)Bazel Version
clinfo
computecpp_info