Compatibility with Rubinius / Ubuntu 14.04

[sam0x17]

Is nmatrix compatible with rubinius? Following the installation guide on the wiki and using Ubuntu 14.04 and rubinius 2.5.2, the following error is occurring for me. When I use ruby 2.2.0, I am able to install nmatrix just fine.

Building native extensions.  This could take a while...
ERROR:  Error installing nmatrix:
    ERROR: Failed to build gem native extension.

    /home/sam/.rvm/rubies/rbx-2.5.2/bin/rbx -r ./siteconf20150524-7120-hufuoo.rb extconf.rb
using C++ standard... c++11
g++ reports version... 4.8.2-19ubuntu1)
checking for main() in -lclapack... no
checking for main() in -llapack... yes
checking for main() in -lcblas... yes
checking for main() in -latlas... yes
checking for atlas/cblas.h... yes
checking for atlas/clapack.h... yes
checking for clapack_dgetrf() in atlas/clapack.h... yes
checking for clapack_dgetri() in atlas/clapack.h... yes
checking for dgesvd_() in clapack.h... yes
checking for cblas_dgemm() in cblas.h... yes
creating nmatrix_config.h
creating Makefile

make "DESTDIR=" clean

make "DESTDIR="
compiling nmatrix.cpp
In file included from data/data.h:46:0,
                 from nmatrix.cpp:58:
data/ruby_object.h:96:9: error: ‘nm::RubyObject::RubyObject(int64_t)’ cannot be overloaded
  inline RubyObject(int64_t other)  : rval(INT2FIX(other)) {}
         ^
data/ruby_object.h:71:9: error: with ‘nm::RubyObject::RubyObject(intptr_t)’
  inline RubyObject(VALUE ref = Qnil) : rval(ref) {}
         ^
In file included from nmatrix.cpp:297:0:
ruby_nmatrix.c: In function ‘intptr_t nm_reshape_bang(intptr_t, intptr_t)’:
ruby_nmatrix.c:1069:11: warning: unused variable ‘elem’ [-Wunused-variable]
     void* elem = s->elements;
           ^
ruby_nmatrix.c: In function ‘SLICE* get_slice(size_t, int, intptr_t*, size_t*)’:
ruby_nmatrix.c:2691:26: error: ‘RHASH_EMPTY_P’ was not declared in this scope
       if (RHASH_EMPTY_P(v)) t++; // go on to the next
                          ^
make: *** [nmatrix.o] Error 1

make failed, exit code 2

[agarie]

Hi Sam! As far as I know, no one ever tried to install or use NMatrix in Rubinius. We developed it with MRI in mind. I'm not familiar with Rubinius to assert how hard it would be, but having NMatrix work with other Ruby implementations seems definitely a good thing.

Can you tell us more about why you want to use NMatrix in Rubinius?

By the way, @wlevine is a Google Summer of Code student working on improving the installation process of NMatrix (among other things) and might be interested in this discussion. :)

[sam0x17]

Hey thanks for reaching out. I am doing some computer vision research and would like to be able to use the easy-to-use syntax of ruby with my research without sacrificing too much performance (in particular, I need support for true parallel threading unlike what MRI has). Rubinius meets my needs in this respect (as does JRuby, but for now it looks like Rubinius is easier to use). I want to use the high performance oily_png library in conjunction with nmatrix to compute some information about a data set containing millions of PNG images.

My understanding is that Rubinius is the closest thing the Ruby community has to a high performance language appropriate for research. Since matrices and linear algebra frequently come up in research-oriented contexts, I think it would be natural for nmatrix to support Rubinius as well as JRuby out of the box.

It would be greatly appreciated if someone could get nmatrix working on the latest Rubinius :)

[sam0x17]

@wlevine would be great if you could work on this it would be much appreciated!

[agarie]

I think @mohawkjohn and @cjfuller will find this discussion interesting as well.

[cjfuller]

My memory of this is hazy, so don't hold me to this, but I think I did actually try to get it to work on a much earlier version of rubinius (years ago), and concluded it was maybe possible. At least back then, their goal was to maintain compatibility for C extensions (haven't checked in on rubinius in a while, so I don't know if this is still true).

Beyond just getting it to compile, one thing that I have (unfortunately) added since then is a bunch of code for marking NMatrix data structures to prevent errant garbage collection. This uses a static C structure and is decidedly not thread-safe. We'd probably need to conditionally compile in locking logic. Shouldn't be too bad so long as rubinius's threading model is something that we can easily deal with from C, but I don't know enough about it to say for sure. (All this also assumes the garbage collection is roughly the same between the two implementations. It could be especially tricky if there's the possibility of it being concurrent with running NMatrix.)

If someone does manage to get it to compile, though, I'm happy to look into the details of thread-safety/gc here.

[brixen]

Hey folks, I can give you some pointers to get this working with Rubinius:

1. The MRI C-API has no standard representation. By that I mean that there is zero separation between the C functions MRI uses to implement Ruby and the C functions that C-exts should use. For this reason, many many C-exts just use whatever C function they find useful at the moment. The problem with this as that numerous MRI implementation details leak into these C-exts.
2. MRI's garbage collector is very different than the Rubinius one. MRI's collector is conservative (or imprecise). This means that the garbage collector doesn't know exactly where all references to objects are and the GC has to "conservatively" treat possible object references as actual references. MRI's GC also does not move objects in memory. In contrast, the Rubinius GC is precise, generational, and does move objects in memory to compact. The consequence is that many C-exts assume the object reference is static and stash references is places the GC cannot see. This is a recipe for a guaranteed SEGV on Rubinius. To avoid this, we hand out handles to objects, not pointers directly. Usually, this is sufficient.
3. Rubinius does not use the same internal structures as MRI. This is typically the biggest source of problems building C-exts on Rubinius. However, the problem is a failure of good C programming practice. Always use function interfaces to objects and never ever assume that you can reach into a structure you don't own. Many MRI C-exts break these rules with things like RSTRING(obj)->ptr, etc. Rubinius implements the core classes in Ruby. Things like Hash have no representation in C code and we don't support most of the RHASH interfaces in MRI.
4. Rubinius has no global interpreter lock, so mutation of shared resources must be synchronized.
5. Finally, Rubinius has a JIT compiler that uses LLVM and can often make Ruby code run very fast (especially if the type profiles are stable and a lot of object creation is not involved). Hence, the need to rely on C-exts is reduced. You may find that a pure Ruby library has better performance on Rubinius than a C-ext does because of all the extra cost of the C-API mechanism we need to use to make up for the very poor interfaces in MRI.

Please do ping us with specific questions if we can help.

[cjfuller]

Awesome, thanks for the detailed pointers! We'll have to do some more investigation before we can figure out how hard this would be (in particular exactly how often we're relying on what the internal structures look like), but this is a great help.

[sam0x17]

By the way, in my use case, I won't be needing to do any synchronization on nmatrix matrices... I'll have a few shared matrices that are used by multiple threads but in a read-only fashion.

On Wed, May 27, 2015 at 1:26 PM, Colin J. Fuller notifications@github.com wrote:

Awesome, thanks for the detailed pointers! We'll have to do some more investigation before we can figure out how hard this would be (in particular exactly how often we're relying on what the internal structures look like), but this is a great help.

Reply to this email directly or view it on GitHub https://github.com/SciRuby/nmatrix/issues/363#issuecomment-106004560.

[translunar]

Not seeing very much demand for this, so closing for now.