Missing optional dependency WARNING

[arijit17]

I installed CNTK 2.6 on Linux, and I get this warning when I import cntk:

Welcome to Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-133-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

3 packages can be updated.
0 updates are security updates.

Last login: Tue Oct  2 06:43:05 2018 from 10.123.101.54
spc@nur-1:~$ source /usr/local/cntk-p35/bin/activate
(cntk-p35) spc@nur-1:~$ python
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cntk as C
/usr/local/cntk-p35/lib/python3.5/site-packages/cntk/cntk_py_init.py:90: UserWarning:

################################################ Missing optional dependency (    MKL     ) ################################################
   CNTK may crash if the component that depends on those dependencies is loaded.
   Visit https://docs.microsoft.com/en-us/cognitive-toolkit/Setup-Linux-Python#mkl for more information.
############################################################################################################################################

  warnings.warn(WARNING_MSG % ('    MKL     ', 'https://docs.microsoft.com/en-us/cognitive-toolkit/Setup-Linux-Python#mkl'))
>>>

How to fix this?

[thiagocrepaldi]

This is just a warning. If you dont use MKL/MKL-DNN, you can just ignore it. If you do, visit https://docs.microsoft.com/en-us/cognitive-toolkit/Setup-Linux-Python#mkl for more information

[thiagocrepaldi]

How did you install CNTK 2.6? pypi.org? Did you download the wheel package? compiled from source? Did you install the CPU or GPU version? Which distro are you using? Could you run ldconfig -p | grep -i mkl and paste the result?

[arijit17]

Hi @thiagocrepaldi: Thanks for your reply.

I simply installed like this:

$ virtualenv --system-site-packages --python=python3.5  /usr/local/cntk-p35
$ source /usr/local/cntk-py35/bin/activate
$ apt-get install openmpi-bin
$ pip3 install cntk-gpu

Here is the result of what you wanted:

$ ldconfig -p | grep -i mkl
        libmkldnn.so.0 (libc6,x86-64) => /usr/local/lib/libmkldnn.so.0
        libmkldnn.so (libc6,x86-64) => /usr/local/lib/libmkldnn.so

BTW: previously I tried installing on Python 3.6 (as Anaconda 5.2 ((Python 2.7 & Python 3.6), 3.6 is enabled by default on my machine[/usr/local/])). Installed CNTK 2.6 via PyPI. Then when I do import cntk, I get error like this #3285

[thiagocrepaldi]

Hi @arijit17 You have MKL-DNN on your path, but not MKL. MKL-DNN depends on MKL, so you need both.

At https://docs.microsoft.com/en-us/cognitive-toolkit/Setup-Linux-Python#mkl, you will find instructions on how to download and uncompress pre-compiled libmklml_gnu.so, libmklml_intel.so and libiomp5.so

[arijit17]

Hi @thiagocrepaldi

I followed this already. Basically, I followed this page. And did these 2 steps: sudo mkdir /usr/local/mklml

sudo wget https://github.com/01org/mkl-dnn/releases/download/v0.14/mklml_lnx_2018.0.3.20180406.tgz
sudo tar -xzf mklml_lnx_2018.0.3.20180406.tgz -C /usr/local/mklml
wget --no-verbose -O - https://github.com/01org/mkl-dnn/archive/v0.14.tar.gz | tar -xzf - && \
cd mkl-dnn-0.14 && \
ln -s /usr/local external && \
mkdir -p build && \
cd build && \
cmake .. && \
make && \
make install && \
cd ../.. && \
rm -rf mkl-dnn-0.14

Output of this looks like this:

[ 61%] Built target test_pooling_backward
[ 62%] Built target test_batch_normalization
[ 64%] Built target test_deconvolution
[ 66%] Built target test_reorder
[ 67%] Built target test_convolution_format_any
[ 69%] Built target test_inner_product_forward
[ 70%] Built target test_pooling_forward
[ 71%] Built target test_inner_product_backward_data
[ 73%] Built target test_convolution_forward_f32
[ 75%] Built target test_iface_pd_iter
[ 76%] Built target test_convolution_forward_s16s16s32
[ 77%] Built target test_convolution_backward_weights_s16s16s32
[ 79%] Built target test_convolution_backward_weights_f32
[ 81%] Built target test_sum
[ 83%] Built target test_convolution_forward_u8s8s32
[ 84%] Built target test_convolution_relu_forward_f32
[100%] Built target benchdnn
Install the project...
-- Install configuration: "Release"
-- Up-to-date: /usr/local/share/doc/mkldnn/LICENSE
-- Installing: /usr/local/lib/libmkldnn.so.0.14.0
-- Up-to-date: /usr/local/lib/libmkldnn.so.0
-- Up-to-date: /usr/local/lib/libmkldnn.so
-- Set runtime path of "/usr/local/lib/libmkldnn.so.0.14.0" to ""
-- Up-to-date: /usr/local/include/mkldnn_debug.h
-- Up-to-date: /usr/local/include/mkldnn.h
-- Up-to-date: /usr/local/include/mkldnn_types.h
-- Up-to-date: /usr/local/include/mkldnn.hpp

But still it doesn't remove the warning.

[thiagocrepaldi]

@arijit17 The compile log you are seeing refers to MKL-DNN. We need to check your MKL (aka MKLML) installation.

List the content of /usr/local/mklml/ and /usr/local/mklml/mklml_lnx_2018.0.3.20180406/. After making sure that is the path for your mklml, do a export LD_LIBRARY_PATH=/usr/local/mklml/mklml_lnx_2018.0.3.20180406/lib/:${LD_LIBRARY_PATH} and try again.

[arijit17]

@thiagocrepaldi Hi. Unfortunately it doesn't help. Just a note: I was using CNTK for a long time. This warning started appearing after the update. Maybe this gives some hints?

[thiagocrepaldi]

@arijit17 This warning was implemented on purpose to help debugging cases where CNTK crashes due to missing dependencies. It is not an error and it can be ignored if you are not using those dependencies.

In order to debug what is going on on your environment, I need you to list the content of /usr/local/lib, /usr/local/mklml/ and /usr/local/mklml/mklml_lnx_2018.0.3.20180406/

[arijit17]

@thiagocrepaldi OK. Here is a list of list of contents:

(cntk-python3) spc@nur-1:/usr/local/lib$ ls
libmkldnn.so                   libopencv_flann.so            libopencv_objdetect.so        libopencv_videoio.so
libmkldnn.so.0                 libopencv_flann.so.3.4        libopencv_objdetect.so.3.4    libopencv_videoio.so.3.4
libmkldnn.so.0.14.0            libopencv_flann.so.3.4.2      libopencv_objdetect.so.3.4.2  libopencv_videoio.so.3.4.2
libopencv_calib3d.so           libopencv_highgui.so          libopencv_photo.so            libopencv_video.so
libopencv_calib3d.so.3.4       libopencv_highgui.so.3.4      libopencv_photo.so.3.4        libopencv_video.so.3.4
libopencv_calib3d.so.3.4.2     libopencv_highgui.so.3.4.2    libopencv_photo.so.3.4.2      libopencv_video.so.3.4.2
libopencv_core.so              libopencv_imgcodecs.so        libopencv_shape.so            libopencv_videostab.so
libopencv_core.so.3.4          libopencv_imgcodecs.so.3.4    libopencv_shape.so.3.4        libopencv_videostab.so.3.4
libopencv_core.so.3.4.2        libopencv_imgcodecs.so.3.4.2  libopencv_shape.so.3.4.2      libopencv_videostab.so.3.4.2
libopencv_dnn.so               libopencv_imgproc.so          libopencv_stitching.so        pkgconfig
libopencv_dnn.so.3.4           libopencv_imgproc.so.3.4      libopencv_stitching.so.3.4    python2.7
libopencv_dnn.so.3.4.2         libopencv_imgproc.so.3.4.2    libopencv_stitching.so.3.4.2  python3.5
libopencv_features2d.so        libopencv_ml.so               libopencv_superres.so
libopencv_features2d.so.3.4    libopencv_ml.so.3.4           libopencv_superres.so.3.4
libopencv_features2d.so.3.4.2  libopencv_ml.so.3.4.2         libopencv_superres.so.3.4.2

(cntk-python3) spc@nur-1:/usr/local/mklml$ ls
mklml_lnx_2018.0.3.20180406

(cntk-python3) spc@nur-1:/usr/local/mklml/mklml_lnx_2018.0.3.20180406$ ls
include  lib  license.txt

(cntk-python3) spc@nur-1:/usr/local/mklml/mklml_lnx_2018.0.3.20180406/lib$ ls
libiomp5.so  libmklml_gnu.so  libmklml_intel.so

And, here is:

 echo $LD_LIBRARY_PATH
/usr/local/mklml/mklml_lnx_2018.0.3.20180406/lib:/usr/local/lib:/usr/local/cuda/lib:/usr/local/cuda/lib64:/usr/local/opencv-3.4.2/build/lib

[thiagocrepaldi]

@arijit17 You have the MKLML in the right folder and your LD_LIBRARY_PATH is correct but ldconfig fails to find it anyway. I have seen this error for some Ubuntu.

Try this:

echo '/usr/local/mklml/mklml_lnx_2018.0.3.20180406/lib' | sudo tee --append /etc/ld.so.conf.d/mklml.conf
sudo ldconfig
ldconfig -p | grep mkl

The output should be something like:

    libmklml_intel.so (libc6,x86-64) => /usr/local/mklml/mklml_lnx_2018.0.3.20180406/lib/libmklml_intel.so
    libmklml_gnu.so (libc6,x86-64) => /usr/local/mklml/mklml_lnx_2018.0.3.20180406/lib/libmklml_gnu.so
    libmkldnn.so.0 (libc6,x86-64) => /usr/local/lib/libmkldnn.so.0
    libmkldnn.so (libc6,x86-64) => /usr/local/lib/libmkldnn.so
    libiomp5.so (libc6,x86-64) => /usr/local/mklml/mklml_lnx_2018.0.3.20180406/lib/libiomp5.so

[arijit17]

Awesome, warning gone @thiagocrepaldi Sorry for troubling on a silly issue :-)

[kaisark]

@arijit17 @thiagocrepaldi I ran into a similar install problem, and changing "make install" to "sudo make install" seemed to resolve the issue...

The https://docs.microsoft.com/en-us/cognitive-toolkit/Setup-Linux-Python#mkl document should be updated to specify "sudo make install".

[recolic]

If I don't have root privilege, how do I update /etc/ld.so.conf.d/mklml.conf? I'm suffering from the problem.

I built cntk from source with cuda, cudnn, cub, nccl, mklml, mkldnn, python. However python test.py still says Missing optional dependency ( MKL ) and Missing optional dependency (GPU-Specific), while import cntk.

# My command
./configure --with-mkl=/home/gpu_ubuntu/recolic/mklml_lnx_2019.0.1.20180928 --with-mkl-sequential=/home/gpu_ubuntu/recolic/mklml_lnx_2019.0.1.20180928 --with-boost=/home/gpu_ubuntu/recolic/.linuxbrew/Cellar/boost/1.68.0 --with-protobuf=/home/gpu_ubuntu/recolic/.linuxbrew/opt/protobuf --with-mpi=/home/gpu_ubuntu/public/software/intel/compilers_and_libraries_2019.1.144/linux/mpi/intel64 --with-swig=/home/gpu_ubuntu/recolic/.linuxbrew/Cellar/swig/3.0.12 --with-py35-path=/home/gpu_ubuntu/recolic/anaconda3/envs/cntk-py35 --with-nccl=/home/gpu_ubuntu/recolic/nccl/build --cuda=yes --with-cuda=/home/gpu_ubuntu/recolic/tmp/cuda-10.0 --with-cub=/home/gpu_ubuntu/recolic/cub-1.8.0 --with-cudnn=/home/gpu_ubuntu/recolic/cudnn-10.0-linux-x64-v7.4.2.24 --with-deps=yes

make -j 32 all

pip install python/cntk*