Error linking cublas

[subversive-owl]

On a Linux laptop with NVIDIA proprietary dependencies installed, all necessary libraries link properly except for cublas.

Note that building with only native extensions works, but examples return immediately without a result in this case (#5).

log

The initial command was cargo build --release.

generated cc command

"cc"
"-Wl,--as-needed"
"-Wl,-z,noexecstack"
"-m64"
"-L"
"/home/aspect/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/juice_examples-9781d21bf85699cb.0.o"
"-o"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/juice_examples-9781d21bf85699cb"
"-Wl,--gc-sections"
"-pie"
"-Wl,-O1"
"-nodefaultlibs"
"-L"
"/home/aspect/projects/spearow/juice-examples/target/release/deps"
"-L"
"/home/aspect/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib"
"-Wl,-Bstatic"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libcsv-2354ff8bb5468373.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libhyper-a53d3e8d4d3fa281.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libpercent_encoding-8930ffe4cbccc723.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libmime-368c7cfe1482830d.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libunicase-f5882c186e2fc601.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libfutures_cpupool-82cc791981ccd882.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libbase64-8dd62d4c27324cbd.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libsafemem-9b8e2350be5abf64.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/liblanguage_tags-4bebcf1b6588e02a.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libjuice-b7e6c2821f098cce.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libcoaster_nn-7c182f364063e3d3.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/librcudnn-bc77ed213bddc6dd.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/librcudnn_sys-7e936b8526305f42.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libcoaster_blas-c193f91bfb80a61f.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/librblas-76d6b0c8c3b1f2a3.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libcoaster-b0d7a3da926aabbe.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/liblazy_static-3391d0e9efc42f3c.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libenum_primitive-3fe05f5b62cd6a07.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libbitflags-ab7d60371fd36f84.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libcapnp-d0d85c39afb38f9d.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libtimeit-139b6c2e5bb2b11d.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libhttparse-4136adfc4bbcabe7.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libtokio_proto-4d8121d02fbd4749.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libtokio_service-961e2dc16f64f437.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libtokio_core-3f9d5f8777e44dc7.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libscoped_tls-e6b554010a211342.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libtake-a3887a0e00a660f6.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libsmallvec-972ec0fed13a0c3e.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libnum-4d84bdfe474beb00.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libnum_rational-5bc9876630a21472.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libnum_bigint-df0daad92318bb8e.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/librand-17d26da4017a015e.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libnum_complex-c835e0cde4414f89.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libnum_iter-20a289d5f33329ec.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libnum_integer-57825ead06c2e49d.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libnum_traits-0b9feee87cc34d99.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libbyteorder-0d2d5bbf440914cd.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libdocopt-31fa79aac4befe23.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libstrsim-6a274aa6277c2545.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/librustc_serialize-ed0b7d2490a131f4.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libregex-bacd721b81398843.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libutf8_ranges-28db1e0329cab7b8.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libregex_syntax-805739da0c85e217.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libaho_corasick-2db5c95bf6c2c91e.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libmemchr-7374d267f95de936.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/librcublas-429ec3243eeb3513.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/librcublas_sys-3b94da837c3d9e0d.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libthread_local-9214de0e7bf1adc3.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libthread_id-541327fc5a4c3cc1.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libmio-d6b382793ac325f7.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libslab-fdda56a9ba794c6f.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libnet2-b5242ae5fbf7008b.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libcfg_if-0a97344db5b79bed.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/liblazycell-53458f46cb6ad3a3.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libtokio_io-0295ffaa56a72968.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libbytes-0f7f419798c4659c.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libiovec-b9bd738d19e90764.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libbyteorder-2361801162d08949.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libfutures-17c2f0d76db207b9.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libnum_cpus-5158227945cdbbdc.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libtime-562bd69e5e6e68ab.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libenv_logger-cade4af5cccedd43.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libregex-e438f9cc5cb4d2d1.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libutf8_ranges-a603754083c5d09f.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libthread_local-d094e7042b6c3e99.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/liblazy_static-2efea79575df2e70.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libunreachable-7d3c981dcc608f3c.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libvoid-bc53f9d88475ace6.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libaho_corasick-d955d245153c4f4d.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libmemchr-e3457abdc9a98596.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/liblibc-0547e85bf76474ea.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/liblog-bc6b8b656776f05e.rlib"
"/home/aspect/projects/spearow/juice-examples/target/release/deps/libregex_syntax-b2bd34b810e9ee57.rlib"
"/home/aspect/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-35ad9950c7e5074b.rlib"
"/home/aspect/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-fb44afc024bbc636.rlib"
"/home/aspect/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-14b8f3202acdad6a.rlib"
"/home/aspect/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librand-20a50a22d4c2b1e9.rlib"
"/home/aspect/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcollections-b479831207997444.rlib"
"/home/aspect/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-10b591f1a68dd370.rlib"
"/home/aspect/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_jemalloc-28913dc5a1e63cd7.rlib"
"/home/aspect/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-6ecacccb5bdc4911.rlib"
"/home/aspect/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_unicode-f4f0ae88f5ad8ad4.rlib"
"/home/aspect/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-bfaa82017ca17cb2.rlib"
"/home/aspect/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-863b57a66ba6c3e1.rlib"
"-Wl,-Bdynamic"
"-l"
"cudnn"
"-l"
"openblas"
"-l"
"OpenCL"
"-l"
"cuda"
"-l"
"cublas"
"-l"
"util"
"-l"
"dl"
"-l"
"rt"
"-l"
"pthread"
"-l"
"gcc_s"
"-l"
"pthread"
"-l"
"c"
"-l"
"m"
"-l"
"rt"
"-l"
"pthread"
"-l"
"util"

errors

note: /usr/bin/ld: cannot find -lcublas
          collect2: error: ld returned 1 exit status

environment

OS

> uname -r
4.12.0-1-amd64

> lsb_release -a
Distributor ID: PureOS
Description:    PureOS GNU/Linux 8
Release:        8
Codename:       green

env vars

env | grep CU                                                 
CUBLAS_INCLUDE_DIR=/usr/local/cuda-8.0/include/            
CUBLAS_LIB_DIR=/usr/local/cuda-8.0/lib64/                  
CUDBLAS_INCLUDE_DIR=/usr/local/cuda-8.0/include            
CUDBLAS_LIB_DIR=/usr/local/cuda-8.0/lib64                  
CUDNN_INCLUDE_DIR=/usr/local/cuda-8.0/include/             
CUDNN_LIB_DIR=/usr/local/cuda-8.0/lib64/                   

offending libraries

❰/usr/local/cuda-8.0❱✔≻ cd lib64
❰/usr/local/cuda-8.0/targets/x86_64-linux/lib❱✔≻ ls -l | grep cublas
lrwxrwxrwx 1 root root        16  6月  6 12:46 libcublas.so -> libcublas.so.8.0
lrwxrwxrwx 1 root root        19  6月  6 12:46 libcublas.so.8.0 -> libcublas.so.8.0.88
-rw-r--r-- 1 root root  42505456  6月  6 12:41 libcublas.so.8.0.61
-rw-r--r-- 1 root root  48454352  6月  6 12:42 libcublas.so.8.0.88
-rw-r--r-- 1 root root  63537142  6月  6 12:41 libcublas_device.a
-rw-r--r-- 1 root root  55059292  6月  6 12:41 libcublas_static.a

[drahnr]

Can you provide a Dockerfile which reproduces the issue? I'd be really happy to dig into this and add it to CI as a test.

[subversive-owl]

I'll have a Dockerfile at the end of the day.

I'm making a docker base image since there is none for PureOS8, which should be ok to upload to the public registry.

[drahnr]

Cool! Concourse CI also works with a rootfs.tar.xz so if you have that, I should be all good too :)

[subversive-owl]

Here's an image: https://hub.docker.com/r/subversiveowl/pureos8-base/

Now I just need to write a dockerfile...

[subversive-owl]

dockerfile provided in https://github.com/spearow/continuous-integration/pull/5 (merged)

[barzamin]

When hitting this issue, consider exporting the following:

export CUBLAS_LIB_DIR=/opt/cuda/lib64
export CUDNN_LIB_DIR=/opt/cuda/lib64

as per [1], [2].