Closed no140 closed 7 years ago
I guess this could go down as 'enhancement', ie 'upgrade tensorflow-cl to 1.0/1.1/latest version'
I'll look into this once stuff is working on the current version :-)
If anyone wants to try this in the meantime, that could be extermely useful :-)
Mostly, changes to upstream tensorflow comprise:
#if GOOGLE_CUDA
, to enable operations. This is fairly trivial to port :-)int32
, bool
and float
on the gpu. In some cases int8
is also allowed. half
and double
are generally removed. int64
is a bit of an open question... minimize ideally, for nowtensorflow/stream_executor/cl
. There are a few tooling changesYou can view the entire set of differences between the current tf-coriander version, and the upstream master it was based on, by going to https://github.com/hughperkins/tf-coriander/compare/orig-master...master and clicking on 'files changes'
meld
or similar to compare themSome additional background on the #if GOOGLE_CUDA
is given in https://github.com/hughperkins/tf-coriander/blob/master/doc/enabling-operations.md
@no140 - I added my experiences with a ~compatible version of Keras to the tf-coriander Wiki: https://github.com/hughperkins/tf-coriander/wiki/Third-Party-Compatibility
So far, most tests pass, though 2 have to be disabled entirely because they segfault in tf-coriander and break the rest. Also, there is a seemingly random tendency for recurrent nets (maybe? Might be unrelated to layer/network type) to "freeze" after behaving normally at first.
Awesome, thanks! :-) . Excellent info on things that are/arent working.
Yea, I reckon using Keras' test-cases might be a useful way to tune tf-coriander at this stage, because Keras is the prototypical consumer of Tensorflow. It works remarkably well! Probably because Keras' abstract backend interface avoids a lot of the murky corners of Tensorflow's API which might work more poorly in Coriander.
One issue I'm having is I don't know how to profile the current used/available memory usage on my GPU (ROCm Driver), so I can't investigate whether there's a memory leak causing the freeze on LSTM training.
Yea, I reckon using Keras' test-cases might be a useful way to tune tf-coriander at this stage, because Keras is the prototypical consumer of Tensorflow. It works remarkably well! Probably because Keras' abstract backend interface avoids a lot of the murky corners of Tensorflow's API which might work more poorly in Coriander.
Cool, sounds good :-)
One issue I'm having is I don't know how to profile the current used/available memory usage on my GPU (ROCm Driver), so I can't investigate whether there's a memory leak causing the freeze on LSTM training.
One option which can give more information, at the expense of sppppaaammmmm is:
ccmake
build configuration:
mkdir build
cd build
ccmake ..
c
to load build configurationCOCL_SPAM
, and some or all of the COCL_SPAM_xxx
optionsc
another couple of times, until g
appears, then press g
to write out hte build configuraitonmake -j 8
python -c 'import tensorflow; import sys; print("\nPlease copy libcocl.so into:\n\n " + sys.modules["tensorflow"].path[0] + "/third_party/coriander\n")'
This will write out a ton of spam... but might give some insight into memory allocations etc.
I'll try this and will run the offending LSTM test cases to get logs, and will share them here. If Coriander requires CUDA to compile though, I won't be able to do so; I'll get back in a few hours with this hopefully. :)
Coriander does not require NVIDIA® CUDA™ Toolkit to compile.
Having great difficulty getting coriander
to compile. Initially, just getting it to find the llvm
libs took a bit of figuring out, because I don't normally work with C(++). But this error seems a bit tricker:
/home/cathal/localapps/coriander/src/new_instruction_dumper.cpp: In member function ‘void cocl::NewInstructionDumper::dumpCall(cocl::LocalValueInfo*, const std::map<llvm::Function*, llvm::Type*>&)’:
/home/cathal/localapps/coriander/src/new_instruction_dumper.cpp:1231:44: error: too few arguments to function ‘llvm::Function* llvm::CloneFunction(const llvm::Function*, llvm::ValueToValueMapTy&, bool, llvm::ClonedCodeInfo*)’
valueMap);
^
In file included from /home/cathal/localapps/coriander/src/new_instruction_dumper.cpp:16:0:
/usr/lib/llvm-3.8/include/llvm/Transforms/Utils/Cloning.h:129:11: note: declared here
Function *CloneFunction(const Function *F, ValueToValueMapTy &VMap,
Any pointers to help solve this? I'm using LLVM 3.8 as installed by Ubuntu.
Ah, nm: I just saw here that it's LLVM4 or nothing: https://github.com/hughperkins/coriander/issues/25#issuecomment-304672757
But, but, but... https://github.com/hughperkins/coriander/blob/master/docker/Dockerfile
? So should this compile in 3.8, or is that Dockerfile outdated?
Dockerfile is outdated
On 5 June 2017 10:46:22 BST, Cathal Garvey notifications@github.com wrote:
But, but, but... https://github.com/hughperkins/coriander/blob/master/docker/Dockerfile
? So should this compile in 3.8, or is that Dockerfile outdated?
-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/hughperkins/tf-coriander/issues/28#issuecomment-306147818
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
If you get a moment to update the Dockerfile, that could be neat :-D
(or you mean, you're not using the Dockerfile, just citing it as hopeful evidence that Coriander can compile in 3.8? :-D )
(updated the README to point out they're updated, and need updating to llvm-4.0 https://github.com/hughperkins/coriander/commit/4fb1f57f16723f939504c69dca0951763c1e5759 )
The latter! But I'm happy to offer PRs when this builds for me locally. I already have suggestions. :)
OK, finally have this built. I'll temporarily install the spam version over my current one, will run the offending Keras tests, and will collect them for inspection. :)
Cool :-) . Sounds good :-)
Hrm hrm hrm.. py.test appears to eat all of the spam output, will need to dig into the PyTest docs a bit.
py.test -svx
the -s
prints stdout. It's horrible though. So, normally I make it runnable directly, with python
, eg see https://github.com/hughperkins/tf-coriander/blob/master/tensorflow/stream_executor/cl/test/test_random.py#L64-L68
if __name__ == '__main__':
if len(sys.argv) == 1:
print('Please run using py.test')
else:
eval('%s((3, 4))' % sys.argv[1])
(in py.test, you can also use -k
to choose one specific test. and you can run a specific test file by path, eg:
py.test -sv tensorflow/stream_executor/cl/test/test_random.py -k truncated
Keras uses a certain amount of custom test magic that I haven't dug into yet, so the simplest thing is to continue using py.test
?
Thanks for the pointers; I've collected the test output for keras-1.1.1/tests/keras/layers/test_recurrent.py
, attached (~15Mb uncompressed.. O_o).
Ok. Seems these are all passing? I searched for error
, case-insensitve, but found nothing?
Those are the tests that pass, but sometimes freeze. So I'm suspicious of memory leaks as a potential cause? But, why it would run OK some times and not others, is mysterious.
I'll open a separate issue in coriander
for the segfaulting tests, shall I? I can provide spammy output from py.test
and the failing kernel in both cases.
The freeze-midway thing also happened when I was doing an embedding + LSTM classification task; trained for plenty of batches/epochs, and then suddenly stopped and simply hung on a certain batch. I was patient but it never recovered after 10m+, so I killed it..
freezing sounds like a loop that never terminates. Probably needs a ton of debugging on my side.
(ie, inside a kernel, there's probably something that loops forever, for some reason...)
right, I'm going to turn off my email, and try to focus on the split star star issue, instead of procrastinating :-P
freezing sounds like a loop that never terminates. Probably needs a ton of debugging on my side.
:/ I'm sorry this isn't something I have the know-how to help with directly!
Thats fair.
So, good/bad news is that autoencoder.py
hangs for me on Mac
(by the way, when I say 'hangs' I mean, my computer keeps running, but I have to do a full Mac restart, before I can run any tensorflow code at all on that gpu :-P , no way of ctrl-C'ing the process )
Hey guys,
What's the status on this? Also, why do you guys have two separate branches for v1.2.1:
v1.2.1-branch & v1.2.1-cl?
Is one of those branches in somewhat working condition?
v1.2.1-branch is the original from upstream. v1.2.1-cl is an attempt to start merging in the opencl stuff.
Neither is in somewhat working condition.
I'm working on other things at the moment. Open opportunity for someone to start looking into this.
What is the strategy of merging with upstream? Every time you must do a lot of work of manual steps? Do you have any ideas how to automate most of the routine work? I want to help, but in my opinion it can be very time consuming if you do it time to time, version to version, especially while tensorflow releases now appear much often.
well... ideally there should be very little to do, since almost all the original work is in coriander. And if you look at the commits, almost all the changes come down to:
#ifdef GOOGLE_CUDA
, eg https://github.com/hughperkins/tf-coriander/commit/a1813a12b2f30b47b7f0c7b38bc77d72357fe6d2 andHowever, there are a few exceptions to this:
tensorflow/stream_executor/cuda
into tensorflow/stream_executor/cl
, https://github.com/hughperkins/tf-coriander/tree/master/tensorflow/stream_executor/cl . I dont know if this is a good idea or not, but it seemed easier at the time...Sorry, I am out of business because of old linux kernel without OpenCL support :(
Fair enough
Hello, on Ubuntu 16.04, installed tensorflow-cl as per instructions in pip3. Keras is version 2.0.5, output error: `
Seen somewhat similar issues online with fix to revert back to tf version 0.10 or upgrade to 0.12.
Has anyone seen this, or successfully used tf-cl in keras (version?)? Simple test importing tensorflow in python (no keras) seem to function okay.