N jobs doesn't affect number of cores used

[austeane]

I downloaded and installed as per instructions, however on my 2017 MacBook Pro running High Sierra, the test script using MNIST always uses 1 core, regardless of the n_jobs parameter passed in.

[informationcake]

I get the same issue with my macbook pro running El Capitan. With verbose=True it states that only 1 core is being used (top confirms this in terminal) regardless of the n_jobs parameter.

[PaulRivaud]

Experiencing the same issue. Is the value hard coded in line 604 there: https://github.com/DmitryUlyanov/Multicore-TSNE/blob/master/multicore_tsne/tsne.cpp ?

[DmitryUlyanov]

It usually means that OpenMP was not found in the system. Although @kernc did a great job by improving compatibility with MacOS, some problems still emerge.

Take a look at this thread https://github.com/DmitryUlyanov/Multicore-TSNE/issues/1 and try the recipes from there. It should help.

[kernc]

Indeed, if you are using a pre-built wheel installed from PyPI with pip install MulticoreTSNE, it likely was built without OpenMP support.

The workaround on OS X is possibly:

brew install libomp
# or alternatively, install Xcode

# Force building the package from source 
pip install -U --no-binary MulticoreTSNE
# or pip install -U git+https://github.com/DmitryUlyanov/Multicore-TSNE.git

---

@DmitryUlyanov Since we are using Homebrew already to install Python in OS X builds, might as well put in there libomp package for releases.

[PaulRivaud]

Thank you the suggestions. I uninstalled MulticoreTSNE using pip, installed libomp and tried building the package from source, but that didn't solve the issue.

[kernc]

When building the source, was this string present?

CMake Warning at CMakeLists.txt:22 (MESSAGE): Not using OpenMP. Performance will suffer.

[PaulRivaud]

Here is the output:

paul@dhcp-24-184:~/Documents$pip install -U git+https://github.com/DmitryUlyanov/Multicore-TSNE.git  
Collecting git+https://github.com/DmitryUlyanov/Multicore-TSNE.git  
  Cloning https://github.com/DmitryUlyanov/Multicore-TSNE.git to /private/tmp/pip-fbXO5V-build  
Collecting numpy (from MulticoreTSNE==0.1)  
  Downloading numpy-1.14.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (4.7MB)
    100% |████████████████████████████████| 4.7MB 256kB/s  
Requirement already up-to-date: cffi in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from MulticoreTSNE==0.1)  

Requirement already up-to-date: pycparser in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from cffi->MulticoreTSNE==0.1)  
Installing collected packages: numpy, MulticoreTSNE  
  Found existing installation: numpy 1.13.3  
    Uninstalling numpy-1.13.3:  
      Successfully uninstalled numpy-1.13.3  
  Running setup.py install for MulticoreTSNE ... done  
Successfully installed MulticoreTSNE-0.1 numpy-1.14.1

Package works fine otherwise but it would be helpful for bigger datasets.

[kernc]

This doesn't print build stage output. Try: pip install --verbose -U --no-deps git+https://github.com/DmitryUlyanov/Multicore-TSNE.git

[sg-s]

I've gotten this working on my MacBook (though I'm using an older version of this repo)

see here: https://github.com/sg-s/Multicore-TSNE

[kernc]

@PaulRivaud Have you managed to build with --verbose flag to determine whether OpenMP is not available and why?

[PaulRivaud]

@kernc Sorry for the late response. I tried using the command but got the following: SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661)

I updated openssl with brew but I still get that error.

[kernc]

SSLError ...

Unrelated to Multicore-TSNE depending on OpenMP. More like something to do with another dependency re-downloaded from PyPI and this recent policy update: https://status.python.org/incidents/hdx7w97m5hr8. The proposed workaround for OS X is to update your pip (pip install -U pip) and, indeed, try again later.

[PaulRivaud]

Thank you, I managed to get rid of the error. Command output with verbose flag:

Collecting https://github.com/DmitryUlyanov/Multicore-TSNE.git
  Looking up "https://github.com/DmitryUlyanov/Multicore-TSNE.git" in the cache
  No cache entry available
  Starting new HTTPS connection (1): github.com
  https://github.com:443 "GET /DmitryUlyanov/Multicore-TSNE.git HTTP/1.1" 301 178
  Updating cache with response from "https://github.com/DmitryUlyanov/Multicore-TSNE.git"
  Caching permanant redirect
  Looking up "https://github.com/DmitryUlyanov/Multicore-TSNE" in the cache
  No cache entry available
  https://github.com:443 "GET /DmitryUlyanov/Multicore-TSNE HTTP/1.1" 200 None
  Downloading https://github.com/DmitryUlyanov/Multicore-TSNE.git
  Downloading from URL https://github.com/DmitryUlyanov/Multicore-TSNE.git
     / 122kB 3.1MB/s  Updating cache with response from "https://github.com/DmitryUlyanov/Multicore-TSNE"

  Cannot unpack file /private/var/folders/sr/dxzg_7ms43z84dfqf9rmnksc0000gn/T/pip-pkFrFJ-unpack/Multicore-TSNE.git (downloaded from /private/var/folders/sr/dxzg_7ms43z84dfqf9rmnksc0000gn/T/pip-5ctuLP-build, content-type: text/html; charset=utf-8); cannot detect archive format
Cleaning up...
Cannot determine archive format of /private/var/folders/sr/dxzg_7ms43z84dfqf9rmnksc0000gn/T/pip-5ctuLP-build
Exception information:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/commands/install.py", line 335, in run
    wb.build(autobuilding=True)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/req/req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/download.py", line 821, in unpack_url
    hashes=hashes
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/download.py", line 663, in unpack_http_url
    unpack_file(from_path, location, content_type, link)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/utils/__init__.py", line 620, in unpack_file
    'Cannot determine archive format of %s' % location
InstallationError: Cannot determine archive format of /private/var/folders/sr/dxzg_7ms43z84dfqf9rmnksc0000gn/T/pip-5ctuLP-build```

[kernc]

Gash, sorry, the command is:

pip install --verbose -U --no-deps git+https://github.com/DmitryUlyanov/Multicore-TSNE.git

(git URL needs git+ prefix).

[PaulRivaud]

Oh okay. So here it is:

    CMake suite maintained and supported by Kitware (kitware.com/cmake).
    -- The CXX compiler identification is AppleClang 7.0.2.7000181
    -- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++
    -- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ -- works
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    -- Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES) (Required is at least version "2")
    -- Could NOT find OpenMP (missing: OpenMP_CXX_FOUND) (Required is at least version "2")
    CMake Warning at CMakeLists.txt:22 (MESSAGE):
      Not using OpenMP.  Performance will suffer.

I will try to look at it.

[kernc]

AppleClang (7.0.2) doesn't seem to ship with OpenMP configured. Two things you can try:

1. Add -fopenmp=libomp to CMakeLists here.
2. Use another compiler, such as GCC.

[vishalmhjn]

Facing the same issue on OSx 10.15