Open TrentBrick opened 6 years ago
I guess you're on Windows, right?
@jnsebgosselin, what's the trick you used to make multiprocessing work there?
Yes. Windows 10
I @TrentBrick, could you share a minimal example that reproduce the issue please? That would be much appreciated.
The code I run is below with all the error messages pasted below that. It loads in a datafile from https://xcelab.net/rm/statistical-rethinking/
Like I said runs with no problems in Jupyter Notebooks. It dies when I try to get the trace (last line of the with statement)
Code:
`# -- coding: utf-8 -- """ Created on Tue Sep 4 20:58:42 2018
@author: fmsft """
import numpy as np import matplotlib.pyplot as plt
'''samples = 10000
res = np.random.normal( np.random.normal(0, 10, samples), np.random.uniform(0,10,samples), samples ) plt.hist(res) plt.show()'''
import os import pandas as pd df = pd.read_csv('rethinking/data/Howell1.csv', sep=';') display(df.head()) display(df.info()) display(df.describe())
df2 = df[df.age >=18]
import pymc3 as pm
basic_model = pm.Model()
X1 = df2.weight Y = df2.height
with basic_model:
# Priors for unknown model parameters
alpha = pm.Normal('alpha', mu=156, sd=100)
beta = pm.Normal('beta', mu=0, sd=10)
sigma = pm.Uniform('sigma', lower=0, upper=50)
# Expected value of outcome
mu = alpha + beta*X1
# Likelihood (sampling distribution) of observations
Y_obs = pm.Normal('Y_obs', mu=mu, sd=sigma, observed=Y)
#step = pm.Slice()
map_estimate = pm.find_MAP(model=basic_model)
print(map_estimate)
start = pm.find_MAP()
step = pm.NUTS()
trace = pm.sample(100, step, start, random_seed=123, progressbar=True)
pm.traceplot(trace)
pm.summary(trace).round(2)
np.random.multivariate_normal(np.array(alpha, beta), covariance)`
`Traceback (most recent call last):
File "
File "D:\Anaconda2\envs\py36\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 678, in runfile execfile(filename, namespace)
File "D:\Anaconda2\envs\py36\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 106, in execfile exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/fmsft/Rethinking Stats/HW4.py", line 56, in
File "D:\Anaconda2\envs\py36\lib\site-packages\pymc3\sampling.py", line 449, in sample trace = _mp_sample(**sample_args)
File "D:\Anaconda2\envs\py36\lib\site-packages\pymc3\sampling.py", line 996, in _mp_sample chain, progressbar)
File "D:\Anaconda2\envs\py36\lib\site-packages\pymc3\parallel_sampling.py", line 275, in init for chain, seed, start in zip(range(chains), seeds, start_points)
File "D:\Anaconda2\envs\py36\lib\site-packages\pymc3\parallel_sampling.py", line 275, in
File "D:\Anaconda2\envs\py36\lib\site-packages\pymc3\parallel_sampling.py", line 182, in init self._process.start()
File "D:\Anaconda2\envs\py36\lib\multiprocessing\process.py", line 105, in start self._popen = self._Popen(self)
File "D:\Anaconda2\envs\py36\lib\multiprocessing\context.py", line 223, in _Popen return _default_context.get_context().Process._Popen(process_obj)
File "D:\Anaconda2\envs\py36\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj)
File "D:\Anaconda2\envs\py36\lib\multiprocessing\popen_spawn_win32.py", line 65, in init reduction.dump(process_obj, to_child)
File "D:\Anaconda2\envs\py36\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj)
BrokenPipeError: [Errno 32] Broken pipe`
The above was formatted weirdly but hope it is sufficient.
Thank you very much, I'll try to take a look at it soon.
I enclosed the entire script in: if __name__ == '__main__':
and that worked for me. Reference: https://discourse.pymc.io/t/multiprocessing-windows-10-brokenpipeerror-errno-32-broken-pipe/2259
@reboss, thanks for letting us know about it! :+1:
Restarting the Kernel works for me. @: Ubuntu 18.04 LTS. Multithreading using the MNE-python parralel_func. (Note: Using all of the available threads).
I am running PyMC3 and it tries to sample my distribution with 4 cores. Every time this results in the above error. When I run this in Jupyter Notebooks instead there is no problem at all. Is there some setting for Spyder that I need to turn on to enable multiprocessing?
I have updated to the most recent versions of Spyder and PyMC3.
Thank you