somaticio / tensorflow.rb

tensorflow for ruby
BSD 3-Clause "New" or "Revised" License
829 stars 63 forks source link

arithmetic on a pointer to void #84

Closed halogenandtoast closed 6 years ago

halogenandtoast commented 7 years ago

When running make I get the following errors:

././files/tf_tensor_helper.cc:43:46: error: arithmetic on a pointer to void
                *(char *)(buffer_data_pointer+i) = file_string[i];

././files/tf_tensor_helper.cc:52:56: error: arithmetic on a pointer to void
                buffer += *(char *)(buffer_data_pointer+i);

././files/tf_tensor_helper.cc:264:39: error: arithmetic on a pointer to void
        auto dst_str = (char *)(cbytes+8);

././files/tf_tensor_helper.cc:274:36: error: arithmetic on a pointer to void
        auto src = (char *)(cbytes + 8);

If I typecast these to char* before the addition I can get the gem to build, but it segfaults when I try to do anything. For instance, here is the backtrace when I try to run the classify_image.rb example:

/Users/matthew-mongeau/.gem/ruby/2.4.0/gems/tensorflow-0.0.1/lib/tensorflow/tensor.rb:59: [BUG] Segmentation fault at 0x0000012ed5a000
ruby 2.4.0p0 (2016-12-24 revision 57164) [x86_64-darwin15]

-- Crash Report log information --------------------------------------------
   See Crash Report log file under the one of following:                    
     * ~/Library/Logs/CrashReporter                                         
     * /Library/Logs/CrashReporter                                          
     * ~/Library/Logs/DiagnosticReports                                     
     * /Library/Logs/DiagnosticReports                                      
   for more details.                                                        
Don't forget to include the above Crash Report log file in bug reports.     

-- Control frame information -----------------------------------------------
c:0005 p:---- s:0044 e:000043 CFUNC  :TF_NewTensor_wrapper
c:0004 p:0280 s:0035 e:000032 METHOD /Users/matthew-mongeau/.gem/ruby/2.4.0/gems/tensorflow-0.0.1/lib/tensorflow/tensor.rb:59 [FINISH]
c:0003 p:---- s:0027 e:000026 CFUNC  :new
c:0002 p:0629 s:0021 E:0010e8 EVAL   classify_image.rb:71 [FINISH]
c:0001 p:0000 s:0003 E:000140 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
classify_image.rb:71:in `<main>'
classify_image.rb:71:in `new'
/Users/matthew-mongeau/.gem/ruby/2.4.0/gems/tensorflow-0.0.1/lib/tensorflow/tensor.rb:59:in `initialize'
/Users/matthew-mongeau/.gem/ruby/2.4.0/gems/tensorflow-0.0.1/lib/tensorflow/tensor.rb:59:in `TF_NewTensor_wrapper'

-- Machine register context ------------------------------------------------
 rax: 0x000000011743f000 rbx: 0x0000000000126000 rcx: 0x000000000008f000
 rdx: 0x0000000000126000 rdi: 0x00000001174d6000 rsi: 0x000000012ed5a000
 rbp: 0x00007fff4fee2830 rsp: 0x00007fff4fee2830  r8: 0x0000000003000001
  r9: 0x0000000000000003 r10: 0x0000000000000000 r11: 0xffffffffe877c000
 r12: 0x00000001102de120 r13: 0x0000000000000004 r14: 0x000000012ecc3000
 r15: 0x000000011743f000 rip: 0x00007fff91a37f49 rfl: 0x0000000000010206

-- C level backtrace information -------------------------------------------
0   ruby                                0x000000010fee72f4 rb_vm_bugreport + 388
1   ruby                                0x000000010fd8166a rb_bug_context + 490
2   ruby                                0x000000010fe62e9f sigsegv + 175
3   libsystem_platform.dylib            0x00007fff91a3552a _sigtramp + 26
4   libsystem_platform.dylib            0x00007fff91a37f49 _platform_memmove$VARIANT$Haswell + 41

-- Other runtime information -----------------------------------------------

* Loaded script: classify_image.rb

* Loaded features:

    0 enumerator.so
    1 thread.rb
    2 rational.so
    3 complex.so
    4 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/x86_64-darwin15/enc/encdb.bundle
    5 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/x86_64-darwin15/enc/trans/transdb.bundle
    6 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/unicode_normalize.rb
    7 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/x86_64-darwin15/rbconfig.rb
    8 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/rubygems/compatibility.rb
    9 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/rubygems/defaults.rb
   10 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/rubygems/deprecate.rb
   11 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/rubygems/errors.rb
   12 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/rubygems/version.rb
   13 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/rubygems/requirement.rb
   14 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/rubygems/platform.rb
   15 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/rubygems/basic_specification.rb
   16 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/rubygems/stub_specification.rb
   17 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/rubygems/util/list.rb
   18 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/x86_64-darwin15/stringio.bundle
   19 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/rubygems/specification.rb
   20 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/rubygems/exceptions.rb
   21 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/rubygems/dependency.rb
   22 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/rubygems/core_ext/kernel_gem.rb
   23 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/monitor.rb
   24 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb
   25 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/rubygems.rb
   26 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/rubygems/path_support.rb
   27 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/gems/2.4.0/gems/did_you_mean-1.1.0/lib/did_you_mean/version.rb
   28 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/gems/2.4.0/gems/did_you_mean-1.1.0/lib/did_you_mean/core_ext/name_error.rb
   29 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/gems/2.4.0/gems/did_you_mean-1.1.0/lib/did_you_mean/levenshtein.rb
   30 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/gems/2.4.0/gems/did_you_mean-1.1.0/lib/did_you_mean/jaro_winkler.rb
   31 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/gems/2.4.0/gems/did_you_mean-1.1.0/lib/did_you_mean/spell_checker.rb
   32 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/2.4.0/delegate.rb
   33 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/gems/2.4.0/gems/did_you_mean-1.1.0/lib/did_you_mean/spell_checkers/name_error_checkers/class_name_checker.rb
   34 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/gems/2.4.0/gems/did_you_mean-1.1.0/lib/did_you_mean/spell_checkers/name_error_checkers/variable_name_checker.rb
   35 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/gems/2.4.0/gems/did_you_mean-1.1.0/lib/did_you_mean/spell_checkers/name_error_checkers.rb
   36 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/gems/2.4.0/gems/did_you_mean-1.1.0/lib/did_you_mean/spell_checkers/method_name_checker.rb
   37 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/gems/2.4.0/gems/did_you_mean-1.1.0/lib/did_you_mean/spell_checkers/null_checker.rb
   38 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/gems/2.4.0/gems/did_you_mean-1.1.0/lib/did_you_mean/formatter.rb
   39 /Users/matthew-mongeau/.rubies/ruby-2.4.0/lib/ruby/gems/2.4.0/gems/did_you_mean-1.1.0/lib/did_you_mean.rb
   40 /Users/matthew-mongeau/.gem/ruby/2.4.0/gems/tensorflow-0.0.1/lib/sciruby/Tensorflow.bundle
   41 /Users/matthew-mongeau/.gem/ruby/2.4.0/gems/tensorflow-0.0.1/lib/tensorflow/opspec.rb
   42 /Users/matthew-mongeau/.gem/ruby/2.4.0/gems/tensorflow-0.0.1/lib/tensorflow/tensor.rb
   43 /Users/matthew-mongeau/.gem/ruby/2.4.0/gems/tensorflow-0.0.1/lib/tensorflow/graph.rb
   44 /Users/matthew-mongeau/.gem/ruby/2.4.0/gems/tensorflow-0.0.1/lib/tensorflow/session.rb
   45 /Users/matthew-mongeau/.gem/ruby/2.4.0/gems/tensorflow-0.0.1/lib/tensorflow/session_options.rb
   46 /Users/matthew-mongeau/.gem/ruby/2.4.0/gems/tensorflow-0.0.1/lib/tensorflow/status.rb
   47 /Users/matthew-mongeau/.gem/ruby/2.4.0/gems/tensorflow-0.0.1/lib/tensorflow/operation.rb
   48 /Users/matthew-mongeau/.gem/ruby/2.4.0/gems/tensorflow-0.0.1/lib/tensorflow/savedmodel.rb
   49 /Users/matthew-mongeau/.gem/ruby/2.4.0/gems/tensorflow-0.0.1/lib/tensorflow/scope.rb
   50 /Users/matthew-mongeau/.gem/ruby/2.4.0/gems/tensorflow-0.0.1/lib/tensorflow/op.rb
   51 /Users/matthew-mongeau/.gem/ruby/2.4.0/gems/tensorflow-0.0.1/lib/tensorflow.rb
arafatkatze commented 7 years ago

@halogenandtoast I am sorry for the issue.
I am very certain that this maybe because you might not be using the latest version of Tensorflow source.
I will try to install everything on a fresh Digital ocean droplet to test the issue.

Thanks

arafatkatze commented 7 years ago

@halogenandtoast I have updated the install script . You can just run this script and it will properly install all the dependencies for Tensorflow and then install tensorflow.rb properly. I have tested it on a fresh ubuntu instance too.
After that you can run bundle exec rake spec for the tests and they will pass.
If you face any issues feel free to comment below.

halogenandtoast commented 7 years ago

I still get the arithmetic issue. I wonder if it has anything to do with me being on OSX and using clang as a compiler instead of GCC?

arafatkatze commented 7 years ago

@halogenandtoast I am almost certain that this might be the reason for your issue.
I am not very well versed with compilers but as of now that is all I can say.

itsmikeq commented 7 years ago

@halogenandtoast - Did you find a way around this issue? Running into the same problem.

halogenandtoast commented 7 years ago

@memikequinn I've had no luck. Any hack I've put in place to get rid of this warning give me a gem that segfaults. I believe the segfault is related to:

[libprotobuf ERROR external/protobuf/src/google/protobuf/io/coded_stream.cc:208] A protocol message was rejected because it was too big (more than 67108864 bytes).  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
arafatkatze commented 7 years ago

@halogenandtoast Even I get a similar message when I do make in the ext directory

./files/tf_tensor_helper.cc: In function ‘std::string tensorflow::String_decoder(TF_Tensor*)’:
./files/tf_tensor_helper.cc:274:38: warning: pointer of type ‘void *’ used in arithmetic [-Wpointer-arith]
         auto src = (char *)(cbytes + 8);

But notice that this is just a warning and not an error.
While writing these functions there was no work around for this and since the intended function was working fine, I chose to ignore the warning.
I think maybe this is an issue with using clang compiler.

halogenandtoast commented 7 years ago

Right, part of the problem is performing arithmetic on a void pointer is undefined behavior which is probably compiler dependent. Is there a reason why these have to be initialized as void pointers? You can always cast them as the correct type (I assume char*) and typecast them to void when needed.

arafatkatze commented 7 years ago

@halogenandtoast That has to do with the tensorflow internals and I can't comment on that.
I am assuming that if they have done it there is a very good reason for it or maybe it was simply unavoidable.

AshtonO commented 7 years ago

@halogenandtoast @Arafatk Was there ever any update/fix for this?

I'm trying to build on Sierra and still running into the same problem.

hakanai commented 7 years ago

Repeating what I said over on #94 which I found first, but fixing the error itself is just a matter of casting before the additional, but I get the segfault as well.

I also get a ton of warnings, any of which could be the cause for the segfault (could also be none of them)... implicit casts from 64-bit to 32-bit types may be it - there's a lot of warnings for those.

msroot commented 6 years ago

Also have the same issue

compiling Tensorflow_wrap.cxx
make iIn file included from Tensorflow_wrap.cxx:2733:
././files/tf_tensor_helper.cc:29:9: warning: cannot delete expression with pointer-to-'void' type 'void *' [-Wdelete-incomplete]
        delete[] data;
        ^        ~~~~
././files/tf_tensor_helper.cc:57:46: error: arithmetic on a pointer to void
                *(char *)(buffer_data_pointer+i) = file_string[i];
                          ~~~~~~~~~~~~~~~~~~~^
././files/tf_tensor_helper.cc:66:56: error: arithmetic on a pointer to void
                buffer += *(char *)(buffer_data_pointer+i);
                                    ~~~~~~~~~~~~~~~~~~~^
././files/tf_tensor_helper.cc:294:39: error: arithmetic on a pointer to void
        auto dst_str = (char *)(cbytes+8);
                                ~~~~~~^
././files/tf_tensor_helper.cc:304:36: error: arithmetic on a pointer to void
        auto src = (char *)(cbytes + 8);
                            ~~~~~~ ^
nst1 warning and 4 errors generated.
make: *** [Tensorflow_wrap.o] Error 1
ioannis@MacBook-Pro tensorflow_c (master) ⚡️  g++ --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 9.0.0 (clang-900.0.38)
Target: x86_64-apple-darwin17.2.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
arafatkatze commented 6 years ago

@halogenandtoast @msroot @trejkaz @memikequinn @AshtonO I have made updates to the code.
Please try the installation process again and hopefully this should work out for you guys.
Feel free to reopen if you have any issues.