luajit -l cltorch -e 'cltorch.test()' passes but th -l cltorch -e 'cltorch.test()' fails

Hi, I'm trying to set up cltorch on OSX el capitan on a macbook pro with AMD GPU.

The tests run and pass using luajit -l cltorch -e 'cltorch.test()' but th -l cltorch -e 'cltorch.test()' fails with the output:

could not load cltorch, skipping
/Users/chris/torch/install/share/lua/5.1/cltorch/init.lua:19: cannot load '/Users/chris/torch/install/lib/lua/5.1/libcltorch.so'
stack traceback:
    [C]: in function 'require'
    /Users/chris/torch/install/share/lua/5.1/cltorch/init.lua:19: in main chunk
    [C]: in function 'require'
    ...hris/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:48: in function <...hris/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:48>
    [C]: in function 'xpcall'
    ...hris/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:48: in main chunk
    [C]: at 0x010ba9cbd0
[string "cltorch.test()"]:1: attempt to index global 'cltorch' (a nil value)

/Users/chris/torch/install/lib/lua/5.1/libcltorch.so definitely exists.

and the output of clinfo is:

Number of platforms                               1
  Platform Name                                   Apple
  Platform Vendor                                 Apple
  Platform Version                                OpenCL 1.2 (Nov 18 2015 20:45:47)
  Platform Profile                                FULL_PROFILE
  Platform Extensions                             cl_APPLE_SetMemObjectDestructor cl_APPLE_ContextLoggingFunctions cl_APPLE_clut cl_APPLE_query_kernel_names cl_APPLE_gl_sharing cl_khr_gl_event

  Platform Name                                   Apple
Number of devices                                 3
  Device Name                                     Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz
  Device Vendor                                   Intel
  Device Vendor ID                                0xffffffff
  Device Version                                  OpenCL 1.2
  Driver Version                                  1.1
  Device OpenCL C Version                         OpenCL C 1.2
  Device Type                                     CPU
  Device Profile                                  FULL_PROFILE
  Max compute units                               8
  Max clock frequency                             2800MHz
  Device Partition                                (core)
    Max number of sub-devices                     0
    Supported partition types                     None
  Max work item dimensions                        3
  Max work item sizes                             1024x1x1
  Max work group size                             1024
  Preferred work group size multiple              1
  Preferred / native vector sizes
    char                                                16 / 16
    short                                                8 / 8
    int                                                  4 / 4
    long                                                 2 / 2
    half                                                 0 / 0        (n/a)
    float                                                4 / 4
    double                                               2 / 2        (cl_khr_fp64)
  Half-precision Floating-point support           (n/a)
  Single-precision Floating-point support         (core)
    Denormals                                     Yes
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
    Correctly-rounded divide and sqrt operations  Yes
  Double-precision Floating-point support         (cl_khr_fp64)
    Denormals                                     Yes
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
    Correctly-rounded divide and sqrt operations  No
  Address bits                                    64, Little-Endian
  Global memory size                              17179869184 (16GiB)
  Error Correction support                        No
  Max memory allocation                           4294967296 (4GiB)
  Unified memory for Host and Device              Yes
  Minimum alignment for any data type             128 bytes
  Alignment of base address                       1024 bits (128 bytes)
  Global Memory cache type                        Read/Write
  Global Memory cache size                        64
  Global Memory cache line                        6291456 bytes
  Image support                                   Yes
    Max number of samplers per kernel             16
    Max size for 1D images from buffer            65536 pixels
    Max 1D or 2D image array size                 2048 images
    Base address alignment for 2D image buffers   1 bytes
    Pitch alignment for 2D image buffers          1 bytes
    Max 2D image size                             8192x8192 pixels
    Max 3D image size                             2048x2048x2048 pixels
    Max number of read image args                 128
    Max number of write image args                8
  Local memory type                               Global
  Local memory size                               32768 (32KiB)
  Max constant buffer size                        65536 (64KiB)
  Max number of constant args                     8
  Max size of kernel argument                     4096 (4KiB)
  Queue properties
    Out-of-order execution                        No
    Profiling                                     Yes
  Profiling timer resolution                      1ns
  Execution capabilities
    Run OpenCL kernels                            Yes
    Run native kernels                            Yes
  Prefer user sync for interop                    Yes
  printf() buffer size                            1048576 (1024KiB)
  Built-in kernels
  Device Available                                Yes
  Compiler Available                              Yes
  Linker Available                                Yes
  Device Extensions                               cl_APPLE_SetMemObjectDestructor cl_APPLE_ContextLoggingFunctions cl_APPLE_clut cl_APPLE_query_kernel_names cl_APPLE_gl_sharing cl_khr_gl_event cl_khr_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_image2d_from_buffer cl_APPLE_fp64_basic_ops cl_APPLE_fixed_alpha_channel_orders cl_APPLE_biased_fixed_point_image_formats cl_APPLE_command_queue_priority

  Device Name                                     Iris Pro
  Device Vendor                                   Intel
  Device Vendor ID                                0x1024500
  Device Version                                  OpenCL 1.2
  Driver Version                                  1.2(Nov 18 2015 20:57:43)
  Device OpenCL C Version                         OpenCL C 1.2
  Device Type                                     GPU
  Device Profile                                  FULL_PROFILE
  Max compute units                               40
  Max clock frequency                             1300MHz
  Device Partition                                (core)
    Max number of sub-devices                     0
    Supported partition types                     None
  Max work item dimensions                        3
  Max work item sizes                             512x512x512
  Max work group size                             512
  Preferred work group size multiple              32
  Preferred / native vector sizes
    char                                                 1 / 1
    short                                                1 / 1
    int                                                  1 / 1
    long                                                 1 / 1
    half                                                 0 / 0        (n/a)
    float                                                1 / 1
    double                                               0 / 0        (n/a)
  Half-precision Floating-point support           (n/a)
  Single-precision Floating-point support         (core)
    Denormals                                     No
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
    Correctly-rounded divide and sqrt operations  Yes
  Double-precision Floating-point support         (n/a)
  Address bits                                    64, Little-Endian
  Global memory size                              1610612736 (1.5GiB)
  Error Correction support                        No
  Max memory allocation                           402653184 (384MiB)
  Unified memory for Host and Device              Yes
  Minimum alignment for any data type             128 bytes
  Alignment of base address                       1024 bits (128 bytes)
  Global Memory cache type                        None
  Image support                                   Yes
    Max number of samplers per kernel             16
    Max size for 1D images from buffer            25165824 pixels
    Max 1D or 2D image array size                 2048 images
    Base address alignment for 2D image buffers   4 bytes
    Pitch alignment for 2D image buffers          32 bytes
    Max 2D image size                             16384x16384 pixels
    Max 3D image size                             2048x2048x2048 pixels
    Max number of read image args                 128
    Max number of write image args                8
  Local memory type                               Local
  Local memory size                               65536 (64KiB)
  Max constant buffer size                        65536 (64KiB)
  Max number of constant args                     8
  Max size of kernel argument                     1024
  Queue properties
    Out-of-order execution                        No
    Profiling                                     Yes
  Profiling timer resolution                      80ns
  Execution capabilities
    Run OpenCL kernels                            Yes
    Run native kernels                            No
  Prefer user sync for interop                    Yes
  printf() buffer size                            1048576 (1024KiB)
  Built-in kernels
  Device Available                                Yes
  Compiler Available                              Yes
  Linker Available                                Yes
  Device Extensions                               cl_APPLE_SetMemObjectDestructor cl_APPLE_ContextLoggingFunctions cl_APPLE_clut cl_APPLE_query_kernel_names cl_APPLE_gl_sharing cl_khr_gl_event cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store cl_khr_image2d_from_buffer cl_khr_gl_depth_images cl_khr_depth_images cl_khr_3d_image_writes

  Device Name                                     AMD Radeon R9 M370X Compute Engine
  Device Vendor                                   AMD
  Device Vendor ID                                0x1021c00
  Device Version                                  OpenCL 1.2
  Driver Version                                  1.2 (Nov 18 2015 20:53:16)
  Device OpenCL C Version                         OpenCL C 1.2
  Device Type                                     GPU
  Device Profile                                  FULL_PROFILE
  Max compute units                               10
  Max clock frequency                             800MHz
  Device Partition                                (core)
    Max number of sub-devices                     0
    Supported partition types                     None
  Max work item dimensions                        3
  Max work item sizes                             256x256x256
  Max work group size                             256
  Preferred work group size multiple              64
  Preferred / native vector sizes
    char                                                 4 / 4
    short                                                2 / 2
    int                                                  1 / 1
    long                                                 1 / 1
    half                                                 0 / 0        (n/a)
    float                                                1 / 1
    double                                               1 / 1        (cl_khr_fp64)
  Half-precision Floating-point support           (n/a)
  Single-precision Floating-point support         (core)
    Denormals                                     No
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               No
    Support is emulated in software               No
    Correctly-rounded divide and sqrt operations  Yes
  Double-precision Floating-point support         (cl_khr_fp64)
    Denormals                                     Yes
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
    Correctly-rounded divide and sqrt operations  No
  Address bits                                    32, Little-Endian
  Global memory size                              2147483648 (2GiB)
  Error Correction support                        No
  Max memory allocation                           536870912 (512MiB)
  Unified memory for Host and Device              No
  Minimum alignment for any data type             128 bytes
  Alignment of base address                       32768 bits (4096 bytes)
  Global Memory cache type                        None
  Image support                                   Yes
    Max number of samplers per kernel             16
    Max size for 1D images from buffer            65536 pixels
    Max 1D or 2D image array size                 2048 images
    Base address alignment for 2D image buffers   256 bytes
    Pitch alignment for 2D image buffers          256 bytes
    Max 2D image size                             16384x16384 pixels
    Max 3D image size                             2048x2048x2048 pixels
    Max number of read image args                 128
    Max number of write image args                8
  Local memory type                               Local
  Local memory size                               32768 (32KiB)
  Max constant buffer size                        65536 (64KiB)
  Max number of constant args                     8
  Max size of kernel argument                     1024
  Queue properties
    Out-of-order execution                        No
    Profiling                                     Yes
  Profiling timer resolution                      37ns
  Execution capabilities
    Run OpenCL kernels                            Yes
    Run native kernels                            No
  Prefer user sync for interop                    Yes
  printf() buffer size                            134217728 (128MiB)
  Built-in kernels
  Device Available                                Yes
  Compiler Available                              Yes
  Linker Available                                Yes
  Device Extensions                               cl_APPLE_SetMemObjectDestructor cl_APPLE_ContextLoggingFunctions cl_APPLE_clut cl_APPLE_query_kernel_names cl_APPLE_gl_sharing cl_khr_gl_event cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store cl_khr_image2d_from_buffer cl_khr_depth_images cl_APPLE_command_queue_priority cl_APPLE_command_queue_select_compute_units cl_khr_fp64

NULL platform behavior
  clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...)  Apple
  clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...)   Success [P0]
  clCreateContext(NULL, ...) [default]            Success [P0]
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU)  Success (1)
    Platform Name                                 Apple
    Device Name                                   Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU)  Success (2)
    Platform Name                                 Apple
    Device Name                                   Iris Pro
    Device Name                                   AMD Radeon R9 M370X Compute Engine
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM)  <checkNullCtxFromType:2073: create context from type CL_DEVICE_TYPE_CUSTOM : error -30>
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL)  Success (3)
    Platform Name                                 Apple
    Device Name                                   Iris Pro
    Device Name                                   AMD Radeon R9 M370X Compute Engine
    Device Name                                   Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz

Hi allthetime, yes, I think it's an rpath issue, which I'm currently trying to fix, but not really succeeding yet.

What is rpath?

cltorch has a module called libcltorch.so, which is installed at ~/torch/install/lib/lua/5.1/libcltorch.so. When it loads, it needs to pull in some other libraries, like libTHCl.dylib, and libclblas.dylib, and a few others. To do this, on other systems it can read the paths from LD_LIBRARY_PATH variable, but this seems not to work very well on Mac OS X, for various reasons, and is considered generally a Bad Idea, apparently.

The other way it can find them is, at build time, the linker stores the locations of the dependencies. It can do this using relative or absolute paths. On MAC OS X, I think it always uses absolute paths (?), but the first bit of such absolute paths can be a special varaible @rpath, which will be replaced at runtime by the current value of @rpath, which is also somehow set by the linker.

Now, I've set up a travis build, and run a few builds, eg https://travis-ci.org/hughperkins/cltorch/builds/113309396 , which all run from .travis.yml file. And in that build, I'm using otool to examine the various paths stored in libcltorch.so, at build time, by the linker. Here's how it looks:

https://travis-ci.org/hughperkins/cltorch/builds/113309396#L1024

0.48s$ otool -L ~/torch/install/lib/lua/5.1/libcltorch.so

/Users/travis/torch/install/lib/lua/5.1/libcltorch.so:
    @rpath/libluaT.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libTHCl.dylib (compatibility version 0.0.0, current version 0.0.0)
    libclBLAS.2.dylib (compatibility version 2.0.0, current version 2.4.0)
    @rpath/libTH.dylib (compatibility version 0.0.0, current version 0.0.0)
    /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1225.1.1)
    libEasyCL.dylib (compatibility version 0.0.0, current version 0.0.0)
    libclew.1.0.0.dylib (compatibility version 1.0.0, current version 0.0.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)

Do you see? libTHCl.dylib uses @rpath, but libclBLAS.2.dylib does not, and it wont be found at runtime, because it's not in ~/torch/install/lib/lua/5.1, it's actually in the parent of the parent, at ~/torch/install/lib. What is @rpath? It can be obatined by using otool too:

https://travis-ci.org/hughperkins/cltorch/builds/113309396#L1037

The command "otool -L ~/torch/install/lib/lua/5.1/libcltorch.so" exited with 0.

0.02s$ otool -l ~/torch/install/lib/lua/5.1/libcltorch.so | grep RPATH -A2
          cmd LC_RPATH
      cmdsize 40
         path @executable_path/../lib (offset 12)
--
          cmd LC_RPATH
      cmdsize 48
         path /Users/travis/torch/install/lib (offset 12)

=> it is ~/torch/install/lib, which is what we want.

So, all the libraries which have @rpath in their location are located. The others are not. If any of them fail to be located, cltorch fails to load.

As far as why it works with luajit -l cltorch, and not with luajit -e 'require 'cltorch' (or the other way around), that's interesting, that's new information, I'm not quite sure yet ;-)

hughperkins / cltorch

luajit -l cltorch -e 'cltorch.test()' passes but th -l cltorch -e 'cltorch.test()' fails #53