Is it very time-consuming for running the example in pystan?

[frenet]

Summary:

Please provide a short couple sentence summary. Is it very time-consuming for running the example in pystan? I've try to run the example in the pystan demo, but after one hours of running, the result is still unknown.

Description:

the pystan was compiling all the ways, I wonder why pystan compiled the code so many times?

Reproducible Steps:

I just tried and tried the example code in https://github.com/stan-dev/pystan, which appeared also in https://pypi.python.org/pypi/pystan .

PyStan Version: pystan-2.14.0.0-cp35-cp35m-win_amd64.whl, downloaded from https://pypi.python.org/pypi/pystan

Python Version: 3.5 64bit

Operating System: win 7 sp1

CPU is i7-4610m 3.00GHz, RAM is 16G. Large space (more than 500G) is available in the hard disk.

[frenet]

pystan example log.docx

[ahartikainen]

Example codes should take only 1 minute or so.

Based on the log, you are compiling code multiple times.

What example are you running and how?

[frenet]

In my laptop, CPU is i7-4610m 3.00GHz, RAM is 16G. Large space (more than 500G) is available in the hard disk. I just tried the example code in https://github.com/stan-dev/pystan, as follows: import pystan import numpy as np import matplotlib.pyplot as plt

schools_code = """ data { int J; // number of schools real y[J]; // estimated treatment effects real sigma[J]; // s.e. of effect estimates } parameters { real mu; real tau; real eta[J]; } transformed parameters { real theta[J]; for (j in 1:J) theta[j] = mu + tau * eta[j]; } model { eta ~ normal(0, 1); y ~ normal(theta, sigma); } """

schools_dat = {'J': 8, 'y': [28, 8, -3, 7, -1, 1, 18, 12], 'sigma': [15, 10, 16, 11, 9, 11, 10, 18]}

fit = pystan.stan(model_code=schools_code, data=schools_dat, iter=1000, chains=4)

print(fit)

eta = fit.extract(permuted=True)['eta'] np.mean(eta, axis=0)

fit.plot() plt.show()

[ahartikainen]

On windows, remember to add njobs=1. Multiprocessing does not currently work.

http://pystan.readthedocs.io/en/latest/windows.html

fit = pystan.stan(model_code=schools_code, data=schools_dat, iter=1000, chains=4, njobs=1)

[frenet]

after I added the parameter njobs=1, then the following shows: del_0fe254f3c524d4153537de9712f3f235, BaseRNG=boost::random::ecuyer1988, Point=stan::mcmc::dense_e_point ] C:\Python35\lib\site-packages\pystan\stan\src\stan/mcmc/hmc/integrators/expl_lea pfrog.hpp(29): note: see reference to function template instantiation 'void stan ::mcmc::base_hamiltonian<Model,stan::mcmc::dense_e_point,BaseRNG>::update_potent ial_gradient(Point &,stan::interface_callbacks::writer::base_writer &,stan::inte rface_callbacks::writer::base_writer &)' being compiled with [ Model=anon_model_0fe254f3c524d4153537de9712f3f235_namespace::anon_mo del_0fe254f3c524d4153537de9712f3f235, BaseRNG=boost::random::ecuyer1988, Point=stan::mcmc::dense_e_point ] C:\Python35\lib\site-packages\pystan\stan\src\stan/mcmc/hmc/hamiltonians/dense_e _metric.hpp(17): note: see reference to class template instantiation 'stan::mcmc ::base_hamiltonian<Model,stan::mcmc::dense_e_point,BaseRNG>' being compiled with [ Model=anon_model_0fe254f3c524d4153537de9712f3f235_namespace::anon_mo del_0fe254f3c524d4153537de9712f3f235, BaseRNG=boost::random::ecuyer1988 ] C:\Python35\lib\site-packages\pystan\stan\src\stan/mcmc/hmc/base_hmc.hpp(130): n ote: see reference to class template instantiation 'stan::mcmc::dense_e_metric<M odel,BaseRNG>' being compiled with [ Model=anon_model_0fe254f3c524d4153537de9712f3f235_namespace::anon_mo del_0fe254f3c524d4153537de9712f3f235, BaseRNG=boost::random::ecuyer1988 ] C:\Python35\lib\site-packages\pystan\stan\src\stan/services/optimize/do_bfgs_opt imize.hpp(47): warning C4267: 'argument': conversion from 'size_t' to 'const int ', possible loss of data C:\Python35\lib\site-packages\pystan\stan_fit.hpp(1213): note: see reference to function template instantiation 'int stan::services::optimize::do_bfgs_optimize< Model,Optimizer,RNG_t,pystan::anonymous-namespace'::PyErr_CheckSignals_Functor> (Model &,BFGSOptimizer &,RNGT &,double &,std::vector<double,std::allocator<_Ty>> &,std::vector<int,std::allocator<int>> &,stan::interface_callbacks::writer::bas e_writer &,stan::interface_callbacks::writer::base_writer &,bool,int,StartIterat ionCallback &)' being compiled with [ Model=anon_model_0fe254f3c524d4153537de9712f3f235_namespace::anon_mo del_0fe254f3c524d4153537de9712f3f235, RNG_t=boost::random::ecuyer1988, BFGSOptimizer=Optimizer, RNGT=boost::random::ecuyer1988, _Ty=double, StartIterationCallback=pystan::anonymous-namespace'::PyErr_CheckSig nals_Functor ] C:\Python35\lib\site-packages\pystan\stan\src\stan/services/optimize/do_bfgs_opt imize.hpp(63): warning C4267: 'argument': conversion from 'size_t' to 'const int ', possible loss of data C:\Python35\lib\site-packages\pystan\stan\src\stan/optimization/newton.hpp(71): warning C4101: 'e': unreferenced local variable C:\Python35\lib\site-packages\pystan\stan_fit.hpp(1317): note: see reference to function template instantiation 'double stan::optimization::newton_step(M &,std::vector<double,std::allocator<_Ty>> &,std::vector<int,std::allocator

&,std::ostream )' being compiled with [ Model=anon_model_0fe254f3c524d4153537de9712f3f235_namespace::anon_mo del_0fe254f3c524d4153537de9712f3f235, M=anon_model_0fe254f3c524d4153537de9712f3f235_namespace::anonmodel 0fe254f3c524d4153537de9712f3f235, _Ty=double ] C:\Python35\lib\site-packages\pystan\stan\src\stan/variational/families/normal_m eanfield.hpp(403): warning C4101: 'e': unreferenced local variable C:\Python35\lib\site-packages\pystan\stan\src\stan/variational/advi.hpp(166): no te: see reference to function template instantiation 'void stan::variational::no rmal_meanfield::calc_grad<Model,BaseRNG>(stan::variational::normal_meanfield &,M &,Eigen::VectorXd &,int,BaseRNG &,stan::interface_callbacks::writer::base_write r &) const' being compiled with [ Model=anon_model_0fe254f3c524d4153537de9712f3f235_namespace::anon_mo del_0fe254f3c524d4153537de9712f3f235, BaseRNG=boost::random::ecuyer1988, M=anon_model_0fe254f3c524d4153537de9712f3f235_namespace::anonmodel 0fe254f3c524d4153537de9712f3f235 ] C:\Python35\lib\site-packages\pystan\stan\src\stan/variational/advi.hpp(149): no te: while compiling class template member function 'void stan::variational::advi <Model,stan::variational::normal_meanfield,RNG_t>::calc_ELBO_grad(const Q &,Q &, stan::interface_callbacks::writer::base_writer &) const' with [ Model=anon_model_0fe254f3c524d4153537de9712f3f235_namespace::anon_mo del_0fe254f3c524d4153537de9712f3f235, RNG_t=boost::random::ecuyer1988, Q=stan::variational::normal_meanfield ] C:\Python35\lib\site-packages\pystan\stan\src\stan/variational/advi.hpp(385): no te: see reference to function template instantiation 'void stan::variational::ad vi<Model,stan::variational::normal_meanfield,RNG_t>::calc_ELBO_grad(const Q &,Q &,stan::interface_callbacks::writer::base_writer &) const' being compiled with [ Model=anon_model_0fe254f3c524d4153537de9712f3f235_namespace::anon_mo del_0fe254f3c524d4153537de9712f3f235, RNG_t=boost::random::ecuyer1988, Q=stan::variational::normal_meanfield ] C:\Python35\lib\site-packages\pystan\stan\src\stan/variational/families/normal_f ullrank.hpp(443): warning C4101: 'e': unreferenced local variable C:\Python35\lib\site-packages\pystan\stan\src\stan/variational/advi.hpp(166): no te: see reference to function template instantiation 'void stan::variational::no rmal_fullrank::calc_grad<Model,BaseRNG>(stan::variational::normal_fullrank &,M & ,Eigen::VectorXd &,int,BaseRNG &,stan::interface_callbacks::writer::base_writer &) const' being compiled with [ Model=anon_model_0fe254f3c524d4153537de9712f3f235_namespace::anon_mo del_0fe254f3c524d4153537de9712f3f235, BaseRNG=boost::random::ecuyer1988, M=anon_model_0fe254f3c524d4153537de9712f3f235_namespace::anonmodel 0fe254f3c524d4153537de9712f3f235 ] C:\Python35\lib\site-packages\pystan\stan\src\stan/variational/advi.hpp(149): no te: while compiling class template member function 'void stan::variational::advi <Model,stan::variational::normal_fullrank,RNG_t>::calc_ELBO_grad(const Q &,Q &,s tan::interface_callbacks::writer::base_writer &) const' with [ Model=anon_model_0fe254f3c524d4153537de9712f3f235_namespace::anon_mo del_0fe254f3c524d4153537de9712f3f235, RNG_t=boost::random::ecuyer1988, Q=stan::variational::normal_fullrank ] C:\Python35\lib\site-packages\pystan\stan\src\stan/variational/advi.hpp(385): no te: see reference to function template instantiation 'void stan::variational::ad vi<Model,stan::variational::normal_fullrank,RNG_t>::calc_ELBO_grad(const Q &,Q & ,stan::interface_callbacks::writer::base_writer &) const' being compiled with [ Model=anon_model_0fe254f3c524d4153537de9712f3f235_namespace::anon_mo del_0fe254f3c524d4153537de9712f3f235, RNG_t=boost::random::ecuyer1988, Q=stan::variational::normal_fullrank ] C:\Python35\lib\site-packages\pystan\stan\lib\stan_math_2.14.0\stan/math/prim/sc al/err/check_greater_or_equal.hpp(24): warning C4267: 'argument': conversion fro m 'size_t' to 'int', possible loss of data C:\Python35\lib\site-packages\pystan\stan\lib\stan_math_2.14.0\stan/math/prim/sc al/err/check_greater_or_equal.hpp(20): note: while compiling class template memb er function 'void stan::math::`anonymous-namespace'::greater_or_equal<T_y,T_low, false>::check(const char ,const char ,const T_y &,const T_low &)' with [ T_y=double, T_low=double ] C:\Python35\lib\site-packages\pystan\stan\lib\stan_math_2.14.0\stan/math/prim/sc al/err/check_greater_or_equal.hpp(83): note: see reference to function template instantiation 'void stan::math::`anonymous-namespace'::greater_or_equal<T_y,T_lo w,false>::check(const char ,const char ,const T_y &,const T_low &)' being comp iled with [ T_y=double, T_low=double ] C:\Python35\lib\site-packages\pystan\stan\lib\stan_math_2.14.0\stan/math/prim/sc al/err/check_greater_or_equal.hpp(83): note: see reference to class template ins tantiation 'stan::math::`anonymous-namespace'::greater_or_equal<T_y,T_low,false> ' being compiled with [ T_y=double, T_low=double ] C:\Python35\lib\site-packages\pystan\stan\lib\stan_math_2.14.0\stan/math/prim/sc al/fun/lb_free.hpp(36): note: see reference to function template instantiation ' void stan::math::check_greater_or_equal<T,TL>(const char ,const char ,const T_ y &,const T_low &)' being compiled with [ T=double, TL=double, T_y=double, T_low=double ] C:\Python35\lib\site-packages\pystan\stan\src\stan/io/writer.hpp(134): note: see reference to function template instantiation 'double stan::math::lb_free<T,doub le>(const T,const TL)' being compiled with [ T=double, TL=double ] C:\Python35\lib\site-packages\pystan\stan\src\stan/io/writer.hpp(133): note: whi le compiling class template member function 'void stan::io::writer::scal ar_lb_unconstrain(double,T &)' with [ T=double ] c:\users\sidg\appdata\local\temp\tmp652phyyo\anon_model_0fe254f3c524d4153537de97 12f3f235.hpp(144): note: see reference to function template instantiation 'void stan::io::writer::scalar_lb_unconstrain(double,T &)' being compiled with [ T=double ] c:\users\sidg\appdata\local\temp\tmp652phyyo\anon_model_0fe254f3c524d4153537de97 12f3f235.hpp(115): note: see reference to class template instantiation 'stan::io ::writer' being compiled C:\Python35\lib\site-packages\pystan\stan\lib\stan_math_2.14.0\stan/math/prim/sc al/err/check_greater_or_equal.hpp(27): warning C4267: 'argument': conversion fro m 'size_t' to 'int', possible loss of data C:\Python35\lib\site-packages\pystan\stan\lib\stan_math_2.14.0\stan/math/prim/ma t/fun/get_base1_lhs.hpp(32): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data c:\users\sidg\appdata\local\temp\tmp652phyyo\anon_model_0fe254f3c524d4153537de97 12f3f235.hpp(348): note: see reference to function template instantiation 'T &st an::math::get_base1_lhs(std::vector<double,std::allocator<_Ty>> &,std::s ize_t,const char ,std::size_t)' being compiled with [ T=double, _Ty=double ] C:\Python35\lib\site-packages\pystan\stan_fit.hpp(1012): note: see reference to function template instantiation 'void anon_model_0fe254f3c524d4153537de9712f3f23 5_namespace::anon_model_0fe254f3c524d4153537de9712f3f235::write_array(RNG &,std::vector<double,std::allocator<_Ty>> &,std::vector<int,std::allocator &,std::vector<_Ty,std::allocator<_Ty>> &,bool,bool,std::ostream ) const' bein g compiled with [ RNG_t=boost::random::ecuyer1988, RNG=boost::random::ecuyer1988, _Ty=double ] C:\Python35\lib\site-packages\pystan\stan\lib\stan_math_2.14.0\stan/math/prim/ma t/fun/get_base1.hpp(32): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data c:\users\sidg\appdata\local\temp\tmp652phyyo\anon_model_0fe254f3c524d4153537de97 12f3f235.hpp(348): note: see reference to function template instantiation 'const T &stan::math::get_base1(const std::vector<double,std::allocator<_Ty>> &,std::size_t,const char ,std::size_t)' being compiled with [ T=double, _Ty=double ] C:\Python35\lib\site-packages\pystan\stan\lib\stan_math_2.14.0\stan/math/prim/sc al/prob/normal_log.hpp(82): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data c:\users\sidg\appdata\local\temp\tmp652phyyo\anon_model_0fe254f3c524d4153537de97 12f3f235.hpp(254): note: see reference to function template instantiation 'stan: :math::var stan::math::normal_log<true,std::vector<stan::math::var,std::allocato r<_Ty>>,int,int>(const T_y &,const T_loc &,const T_scale &)' being compiled with [ _Ty=stan::math::var, T_y=std::vector<stan::math::var,std::allocator>, T_loc=int, T_scale=int ] C:\Python35\lib\site-packages\pystan\stan\lib\stan_math_2.14.0\stan/math/prim/sc al/prob/normal_log.hpp(84): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data C:\Python35\lib\site-packages\pystan\stan\lib\stan_math_2.14.0\stan/math/prim/sc al/prob/normal_log.hpp(88): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data C:\Python35\lib\site-packages\pystan\stan\lib\stan_math_2.14.0\stan/math/prim/sc al/prob/normal_log.hpp(89): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data C:\Python35\lib\site-packages\pystan\stan\lib\stan_math_2.14.0\stan/math/prim/sc al/prob/normal_log.hpp(107): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data C:\Python35\lib\site-packages\pystan\stan\lib\stan_math_2.14.0\stan/math/prim/sc al/prob/normal_log.hpp(109): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data C:\Python35\lib\site-packages\pystan\stan\lib\stan_math_2.14.0\stan/math/prim/sc al/prob/normal_log.hpp(111): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data stanfit4anon_model_0fe254f3c524d4153537de9712f3f235_3680263616119857559.obj : wa rning LNK4197: export 'PyInit_stanfit4anon_model_0fe254f3c524d4153537de9712f3f23 5_3680263616119857559' specified multiple times; using first specification Creating library C:\Users\sidg\AppData\Local\Temp\tmp652phyyo\Users\sidg\AppD ata\Local\Temp\tmp652phyyo\stanfit4anon_model_0fe254f3c524d4153537de9712f3f235_3 680263616119857559.cp35-win_amd64.lib and object C:\Users\sidg\AppData\Local\Tem p\tmp652phyyo\Users\sidg\AppData\Local\Temp\tmp652phyyo\stanfit4anon_model_0fe25 4f3c524d4153537de9712f3f235_3680263616119857559.cp35-win_amd64.exp Generating code Finished generating code Traceback (most recent call last): File "", line 1, in File "C:\Python35\lib\site-packages\pystan\api.py", line 385, in stan raise ValueError("Parameter {} is not recognized.".format(arg)) ValueError: Parameter njobs is not recognized.

[ahartikainen]

Sorry, typo njobs ->n_jobs

fit = pystan.stan(model_code=schools_code, data=schools_dat, iter=1000, chains=4, n_jobs=1)

[bob-carpenter]

What does "the pystan was compiling all the ways, I wonder why pystan compiled the code so many times?" mean? What makes you think the code's being compiled multiple times? Are you seeing any disk thrashing or is everything running in memory.

[frenet]

I can see 4 cl.exe process running in my task manager, and the log output by the pystan is copied from eclipse log window, which is uploaded as "pystan example log.docx" mentioned above.

[frenet]

Thank you all. Finally I successfully run the example in pystan.

[ariddell]

Glad to hear it worked!