Doesn't compile on OSX

[wesrer]

Hi, I am trying to compile from source, but I get a bunch of errors for seemingly missing files. Here's my output for rake compile --trace

** Invoke compile (first_time)
** Invoke compile:universal.x86_64-darwin18 (first_time)
** Invoke compile:nmatrix:universal.x86_64-darwin18 (first_time)
** Invoke copy:nmatrix:universal.x86_64-darwin18:2.3.7 (first_time)
** Invoke lib/ (first_time, not_needed)
** Invoke tmp/universal.x86_64-darwin18/nmatrix/2.3.7/nmatrix.bundle (first_time)
** Invoke tmp/universal.x86_64-darwin18/nmatrix/2.3.7/Makefile (first_time, not_needed)
** Invoke tmp/universal.x86_64-darwin18/nmatrix/2.3.7 (first_time, not_needed)
** Invoke ext//extconf.rb (first_time, not_needed)
** Invoke ext//blas.c (first_time, not_needed)
** Invoke ext//lapack.c (first_time, not_needed)
** Invoke ext//ruby_nmatrix.c (first_time, not_needed)
** Invoke ext//sparse.c (first_time, not_needed)
** Invoke ext//statistics.c (first_time, not_needed)
** Execute tmp/universal.x86_64-darwin18/nmatrix/2.3.7/nmatrix.bundle
cd tmp/universal.x86_64-darwin18/nmatrix/2.3.7
/usr/bin/make
compiling ../../../../ext/ruby_nmatrix.c
../../../../ext/ruby_nmatrix.c:3:10: fatal error: 'cblas.h' file not found
#include "cblas.h"
         ^~~~~~~~~
1 error generated.
make: *** [ruby_nmatrix.o] Error 1
rake aborted!
Command failed with status (2): [/usr/bin/make...]
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/file_utils.rb:67:in `block in create_shell_runner'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/file_utils.rb:57:in `sh'
/Library/Ruby/Gems/2.3.0/gems/rake-compiler-0.9.9/lib/rake/extensiontask.rb:157:in `block (2 levels) in define_compile_tasks'
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/fileutils.rb:128:in `chdir'
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/fileutils.rb:128:in `cd'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/file_utils_ext.rb:38:in `chdir'
/Library/Ruby/Gems/2.3.0/gems/rake-compiler-0.9.9/lib/rake/extensiontask.rb:156:in `block in define_compile_tasks'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:237:in `block in invoke_prerequisites'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:235:in `each'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:235:in `invoke_prerequisites'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:212:in `block in invoke_with_call_chain'
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:237:in `block in invoke_prerequisites'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:235:in `each'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:235:in `invoke_prerequisites'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:212:in `block in invoke_with_call_chain'
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:237:in `block in invoke_prerequisites'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:235:in `each'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:235:in `invoke_prerequisites'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:212:in `block in invoke_with_call_chain'
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:237:in `block in invoke_prerequisites'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:235:in `each'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:235:in `invoke_prerequisites'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:212:in `block in invoke_with_call_chain'
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/application.rb:160:in `invoke_task'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `each'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block in top_level'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/application.rb:125:in `run_with_threads'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/application.rb:110:in `top_level'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/application.rb:83:in `block in run'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/application.rb:186:in `standard_exception_handling'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/application.rb:80:in `run'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/rake:22:in `load'
/usr/local/bin/rake:22:in `<main>'
Tasks: TOP => compile => compile:universal.x86_64-darwin18 => compile:nmatrix:universal.x86_64-darwin18 => copy:nmatrix:universal.x86_64-darwin18:2.3.7 => tmp/universal.x86_64-darwin18/nmatrix/2.3.7/nmatrix.bundle

and here's my output for rake test --trace

** Invoke test (first_time)
** Execute test
/Users/elyah/Sayak/Github/numruby/lib/nmatrix/nmatrix.rb:39: warning: shadowing outer local variable - j
/Users/elyah/Sayak/Github/numruby/lib/nmatrix/nmatrix.rb:71: warning: shadowing outer local variable - j
/Users/elyah/Sayak/Github/numruby/lib/nmatrix/lapack.rb:12: warning: assigned but unused variable - m
/Users/elyah/Sayak/Github/numruby/lib/nmatrix/lapack.rb:12: warning: assigned but unused variable - n
/Users/elyah/Sayak/Github/numruby/lib/nmatrix/lapack.rb:81: warning: assigned but unused variable - lu
/Users/elyah/Sayak/Github/numruby/lib/nmatrix/lapack.rb:81: warning: assigned but unused variable - ipiv

File does not exist: /Users/elyah/Sayak/Github/numruby/lib/nmatrix.so

rake aborted!
Command failed with status (1): [ruby -w -I"lib:test:lib" -I"/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib" "/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/rake_test_loader.rb" "test/blas_test.rb" "test/elementwise_test.rb" "test/lapack_test.rb" "test/nmatrix_test.rb" "test/numruby_test.rb" "test/statistics_test.rb" ]
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/testtask.rb:130:in `block (3 levels) in define'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/file_utils.rb:57:in `sh'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/file_utils.rb:105:in `ruby'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/testtask.rb:117:in `block (2 levels) in define'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/file_utils_ext.rb:59:in `verbose'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/testtask.rb:111:in `block in define'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/application.rb:160:in `invoke_task'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `each'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block in top_level'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/application.rb:125:in `run_with_threads'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/application.rb:110:in `top_level'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/application.rb:83:in `block in run'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/application.rb:186:in `standard_exception_handling'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/lib/rake/application.rb:80:in `run'
/Library/Ruby/Gems/2.3.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/rake:22:in `load'
/usr/local/bin/rake:22:in `<main>'
Tasks: TOP => test

[Uditgulati]

You need to install BLAS and LAPACK libraries first. You need to install corresponding libraries to libopenblas and liblapack for OSX. Install steps for debian based systems is at: https://github.com/SciRuby/numruby/blob/e52b49394fa0aaf575284ff6278cbb68602ad48c/travis.sh#L26

Try the answers in this link https://stackoverflow.com/questions/38114201/installing-lapack-and-blas-libraries-for-c-on-mac-os and post output logs here please.

[developerfab]

For OSx Catalina, I have installed libopenblasp with brew install openblas and for libblapack with brew install lapack

but this doesn't works for me, I think I should add some symbolic link but I'm not sure 🤔 , @wesrer have you solved this problem?

Sources:

1. https://gist.github.com/zachmayer/e591cf868b3a381a01d6

[Uditgulati]

@developerfab is correct. Both openblas and lapack and not properly symlinked to /usr/local. If you do brew info openblas, you get something like this:

openblas: stable 0.3.0 (bottled), HEAD [keg-only]
Optimized BLAS library
https://www.openblas.net/
/usr/local/Cellar/openblas/0.3.0 (22 files, 139MB)
  Poured from bottle on 2018-05-31 at 20:42:55
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/openblas.rb
==> Dependencies
Required: gcc ✔
==> Options
--with-openmp
        Enable parallel computations with OpenMP
--HEAD
        Install HEAD version
==> Caveats
This formula is keg-only, which means it was not symlinked into /usr/local,
because macOS provides BLAS and LAPACK in the Accelerate framework.

So, one way is to either forcefully link openblas and lapack to /usr/local and other is to just add /usr/opt/ paths of openblas and lapack to compilation step, so that they can be linked through that.

I've made a PR which does this and can be found here at https://github.com/SciRuby/numruby/pull/40.

@wesrer @developerfab please try this fix and let me know if it fixes the issue on your machine. I got it to work on a friend's mac.

[developerfab]

I added this in my ~/.bash_profile I'm not sure if it is necessary :thinking: :

export LDFLAGS="-L/usr/local/opt/lapack/lib"
export LDFLAGS="$LDFLAGS:-L/usr/local/opt/openblas/lib"
export CPPFLAGS="-I/usr/local/opt/lapack/include"
export CPPFLAGS="$CPPFLAGS:-I/usr/local/opt/openblas/include"
export PKG_CONFIG_PATH="/usr/local/opt/lapack/lib/pkgconfig"
export PKG_CONFIG_PATH="$PKG_CONFIG_PATH:/usr/local/opt/openblas/lib/pkgconfig"

The output when I ran the rake compile was:

64 warnings generated.
linking shared-object nmatrix.bundle
ld: warning: directory not found for option '-L/opt/local/lib'
cd -
mkdir -p tmp/x86_64-darwin19/stage/lib/
install -c tmp/x86_64-darwin19/nmatrix/2.6.5/nmatrix.bundle lib//nmatrix.bundle
cp tmp/x86_64-darwin19/nmatrix/2.6.5/nmatrix.bundle tmp/x86_64-darwin19/stage/lib//nmatrix.bundle

I think that this ran perfectly.

[Uditgulati]

@developerfab this pretty much does the same thing as the PR I submitted. Both provide location for openBLAS and LAPACK so that the compiler can link them. But this approach requires user to manually add the path and is also on system level and not project level. So, I believe the PR solution should be a better one.

Also, let me know if it works only with the change done in PR and without changing the ‘ ~/.bash_profile’.

[iamrajiv]

@wesrer Have you solved this issue? I am getting this error when I run rake test

➜  numruby git:(master) rake test
/Users/rajivranjansingh/Desktop/numruby/lib/numruby/lapack.rb:12: warning: assigned but unused variable - m
/Users/rajivranjansingh/Desktop/numruby/lib/numruby/lapack.rb:12: warning: assigned but unused variable - n
/Users/rajivranjansingh/Desktop/numruby/lib/numruby/lapack.rb:81: warning: assigned but unused variable - lu
/Users/rajivranjansingh/Desktop/numruby/lib/numruby/lapack.rb:81: warning: assigned but unused variable - ipiv

File does not exist: /Users/rajivranjansingh/Desktop/numruby/lib/numruby.so

rake aborted!
Command failed with status (1)
/Library/Ruby/Gems/2.6.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => test
(See full trace by running task with --trace)