search

uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.38k stars 89 forks source link

ModuleNotFoundError: No module named 'pathos.multiprocessing' #149

Closed rkatriel-mdsol closed 6 years ago

rkatriel-mdsol commented 6 years ago

Hi,

I installed pathos (using pip3.6) on an Ubuntu Linux instance in AWS but am getting the following error when trying to run a Python3.6 program that works fine on another (virtually identical) Linux instance:

ModuleNotFoundError: No module named 'pathos.multiprocessing'

I found a few references to this issue by googling the error message but no real solution. Pathos appears to be installed correctly (see the trace below).

Looking forward to your suggestions.

Thanks, Ron

$ sudo -H pip3.6 install pathos Requirement already satisfied: pathos in /usr/local/lib/python3.6/dist-packages (0.2.2.1) Requirement already satisfied: pox>=0.2.4 in /usr/local/lib/python3.6/dist-packages (from pathos) (0.2.4) Requirement already satisfied: ppft>=1.6.4.8 in /usr/local/lib/python3.6/dist-packages (from pathos) (1.6.4.8) Requirement already satisfied: dill>=0.2.8.2 in /usr/local/lib/python3.6/dist-packages (from pathos) (0.2.8.2) Requirement already satisfied: multiprocess>=0.70.6.1 in /usr/local/lib/python3.6/dist-packages (from pathos) (0.70.6.1) Requirement already satisfied: six>=1.7.3 in /usr/lib/python3/dist-packages (from ppft>=1.6.4.8->pathos) (1.10.0)

rkatriel-mdsol commented 6 years ago

Update: This issue is specific to the python version installed using Linuxbrew. I installed anaconda and the issue went away. In case it matters, the former uses GCC 5.4.0 while the latter was build with [GCC 7.2.0].

mmckerns commented 6 years ago

First off, is this still an issue for you? If not, you can close it. Otherwise, I'll need more information than the above. It would appear that pathos is installed. So the next question would be could you import pathos? And if so, what do you see when you do dir(pathos)?

rkatriel-mdsol commented 6 years ago

Hi Mike. This is not an issue anymore. Installing the Anaconda version of python solved the issue.

Regarding your debugging request, below is the trace from the version of python that does not work. Immediately below it is the trace from the Anaconda version that works. Comparing both, it looks like the modules were not loaded into the default version of python.

Thanks, Ron

$ /usr/bin/python3.6

Python 3.6.6 (default, Jun 28 2018, 04:42:43)

[GCC 5.4.0 20160609] on linux

Type "help", "copyright", "credits" or "license" for more information.

import pathos

dir(pathos)

['doc', 'loader', 'name', 'package', 'path', 'spec']

$ python3.6

Python 3.6.3 |Anaconda, Inc.| (default, Oct 13 2017, 12:02:49)

[GCC 7.2.0] on linux

Type "help", "copyright", "credits" or "license" for more information.

import pathos

dir(pathos)

['SCP_Launcher', 'SSH_Launcher', 'SSH_Tunnel', 'author', 'builtins', 'cached', 'doc', 'file', 'license', 'loader', 'name', 'package', 'path', 'spec', 'version', 'abstract_launcher', 'citation', 'connection', 'core', 'helpers', 'hosts', 'info', 'license', 'logger', 'multiprocessing', 'parallel', 'pools', 'pp', 'python', 'secure', 'selector', 'serial', 'server', 'threading', 'util']

On August 31, 2018 at 9:53:19 AM, Mike McKerns (notifications@github.com) wrote:

First off, is this still an issue for you? If not, you can close it. Otherwise, I'll need more information than the above. It would appear that pathos is installed. So the next question would be could you import pathos? And if so, what do you see when you do dir(pathos)?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/uqfoundation/pathos/issues/149#issuecomment-417671193, or mute the thread https://github.com/notifications/unsubscribe-auth/ADcH5HfNQwVVQ42p7xLFMxAAYRbclRbGks5uWT_NgaJpZM4WT0n6 .

mmckerns commented 6 years ago

@rkatriel-mdsol: That's weird... I don't think I've seen something like that, where the modules are missing. Maybe it's a bad build... I don't know why that would happen. However, since you have a resolution, I'm going to close this. I'll assume if someone else has this same issue, they will comment here.

Redysz commented 5 years ago

@mmckerns I have the same problem. Version downloaded by pip: 0.2.4. Code:

from pathos.multiprocessing import ProcessPool as Pool
import pathos

print(dir(pathos))
p = Pool(8)

gives me:

['SCP_Launcher', 'SSH_Launcher', 'SSH_Tunnel', '__author__', '__builtins__', '__cached__', '__doc__', '__file__', '__license__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', 'abstract_launcher', 'citation', 'connection', 'core', 'helpers', 'hosts', 'info', 'license', 'logger', 'multiprocessing', 'parallel', 'pools', 'pp', 'python', 'secure', 'selector', 'serial', 'server', 'threading', 'util']
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'multiprocess'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'multiprocess'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'multiprocess'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'multiprocess'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'multiprocess'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'multiprocess'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'multiprocess'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'multiprocess'

There is one difference: multiprocess and multiprocessing. Packages which I have installed (by pip): dill, multiprocess, pathos, pip, pox, ppft, setuptools, six

mmckerns commented 5 years ago

@Redysz: Sorry about the slow reply. I'm not sure it's the same issue, yours looks like a different, more common, install/build error. I don't know what your OS is or version of python, but when you install multiprocess, if you don't have a C compiler, or don't use the wheel (on windows) -- this can happen. It can also happen if you have multiple versions of python installed (i.e. with brew and conda) or have some other PATH related issue. I'd need to know more about your case. But first I should ask, have you already resolved this for yourself @Redysz?

carcigenicate commented 4 years ago

@mmckerns I have the same problem. Version downloaded by pip: 0.2.4. Code:

from pathos.multiprocessing import ProcessPool as Pool
import pathos

print(dir(pathos))
p = Pool(8)

gives me:

['SCP_Launcher', 'SSH_Launcher', 'SSH_Tunnel', '__author__', '__builtins__', '__cached__', '__doc__', '__file__', '__license__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', 'abstract_launcher', 'citation', 'connection', 'core', 'helpers', 'hosts', 'info', 'license', 'logger', 'multiprocessing', 'parallel', 'pools', 'pp', 'python', 'secure', 'selector', 'serial', 'server', 'threading', 'util']
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'multiprocess'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'multiprocess'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'multiprocess'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'multiprocess'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'multiprocess'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'multiprocess'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'multiprocess'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'multiprocess'

There is one difference: multiprocess and multiprocessing. Packages which I have installed (by pip): dill, multiprocess, pathos, pip, pox, ppft, setuptools, six

I had this problem. It seems to be a PATH issue.

I had run pip install pathos inside the PyCharm terminal, and got this error. I was able to fix it by running pip install pathos in the windows terminal as well. Not sure exactly what the problem is, but at least it's working now; although I think I have two copies installed now.