Closed davidsteinar closed 3 years ago
@twiecki I got a working Windows conda env for pymc3.
First by installing MinGW and following these installation instructions, setting minwg bin folder to my Path environment variable.
Then I did the following:
git clone https://github.com/pymc-devs/pymc3.git
cd pymc3
cd conda-envs
conda create env -f environment-dev-py38.yml
conda activate environment-dev-py38
conda install -c conda-forge pymc3
Cygwin was producing errors in the compilation step.
Is that different from the windows installation instructions? Should we update them? CC @michaelosthege
The Windows job in the CI pipelines for v3 also started failing recently. It's blocking PRs such as #4745 and #4741. If you can pinpoint which dependency is incompatible and mark that in the requirements and environment YAML files that would be best! We should not make everybody rely on our installation instructions, but instead anti-pin the I compatible dependencies so the pip/conda package managers take care of it.
For v4 the Windows job is still disabled, by the way 😬
@twiecki Yes the steps I posted are different from the windows installation instruction. The current windows installation instruction also does not mention the requirement of MinGW.
@michaelosthege The compilation error stems from lib\site-packages\theano\
. I am not sure how I can help further, but both environments are using theano-pymc 1.1.2 py38h885f38d_0 conda-forge
I just confirmed that after installing with pip
I don't get any problems.
After pip uninstall pymc3
followed by conda install -c conda-forge pymc3
it also runs fine.
However
WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
in both cases.FWIW I don't trust conda-forge when it comes to PyMC3.
I do have Visual Studio installed, so maybe that gives me a headstart when it comes to compilation stuff.
@michaelosthege maybe mingw is the safer option?
When I do
conda create env -f environment-dev-py38.yml
conda activate environment-dev-py38
pip install pymc3
I do not get any compilation error, so that works too.
However I always get the WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
in any environment.
I also have Visual Studio installed, but not with any specific C++ components since I do not work with C++
Our install instructions include m2w64-toolchain
. Correct me if I'm wrong, but that is MinGW, right?
We should specify OS-dependent dependencies in our requirements/setup.py/conda recipes. I have no idea how, but there's probably a way.
Okay, getting the BLAS stuff installed properly is really hard..
The sunode
installation steps for Windows worked for me.
Apparently sunode
is already doing a better job with the dependencies in its conda recipe.
Don't ask me why m2w64-toolchain mkl-service libpython
are installed in a second step from the default channel. When I wrote those install instructions I tested countless combinations and orders of installation and this is what I ended up with.
On the plus side, by following the sunode
instructions you'll get blazingly fast ODE solvers.
Getting ~Windows~ non-Linux install instructions right is unfortunately a never ending story.
With the sunode installation I do not get the BLAS warning so that seems to be working, but still get the Error: invalid register for .seh_savexmm\r. '
error.
To get a working truly working environment I turned to the official docker image,
docker pull pymc/pymc3
docker run --rm -d -p 8888:8888/tcp pymc/pymc3:latest
docker exec -it 054e90a4789ba0b562dc8837611239c208694b162fed4b0953d15071bb6bef1d bash
(base) jovyan@054e90a4789b:~$ python
Python 3.9.2 | packaged by conda-forge | (default, Feb 21 2021, 05:02:46)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymc3
You are running the v4 development version of PyMC3 which currently still lacks key features. You probably want to use the stable v3 instead which you can either install via conda or find on the v3 GitHub branch: https://github.com/pymc-devs/pymc3/tree/v3
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/jovyan/pymc3/__init__.py", line 43, in <module>
__set_compiler_flags()
File "/home/jovyan/pymc3/__init__.py", line 37, in __set_compiler_flags
import aesara
ModuleNotFoundError: No module named 'aesara'
but that also produces errors ......
The latest
Docker image builds from master
/main
and as you can see we're not testing it much..
But try the following. This just worked for me and this time I confirmed with the getting started notebook too. (Not sure if I did that for my last comment last night.) These are install instructions we recently wrote for the SI of a paper.. So I sincerely hope they are reproducible.
conda create -f environment.yml
conda activate pm3env
conda install m2w64-toolchain
When I do
conda create -f environment.yml
conda activate pm3env
conda install m2w64-toolchain
I get the same error Error: invalid register for .seh_savexmm\r. '
However, when do
conda create -f environment.yml
conda activate pm3env
I get a working env, with BLAS enabled and no compilation error.
Not sure if that's good or bad. Congrats to your working environment 👍
I added a warning to the wiki page and also linked to the comments above.
I just updated the installation guide with a minimal version.
It turned out that numba
and scipy
should be in the dependencies at the same time. Either one individually is not enough..
Getting this error on Windows 10 after following the guide exactly. Have tried many suggestions to fix this problem and even completely uninstalled Anaconda and re-installed to start from scratch. Any help appreciated! Let me know if any more information would be useful in resolving this problem.
Exception: ('Compilation failed (return status=1): C:\\Users\\lswad\\AppData\\Local\\Temp\\ccPHLGwt.s: Assembler messages:\r. C:\\Users\\lswad\\AppData\\Local\\Temp\\ccPHLGwt.s:2324: Error: invalid register for .seh_savexmm\r. ', 'FunctionGraph(Elemwise{EQ}(<TensorType(float64, row)>, <TensorType(int8, (True, True))>))')
@edt-andrew I just tried with the conda create
commands but couldn't replicate the problem.
I'm on Windows 10 with miniconda. Do you maybe have Cygwin or something like that installed? A quick Google search returned some results mentioning it. I don't have Cygwin on my system. Maybe that's the difference?
I installed MinGW after seeing it suggested as a solution but didn't have that initially installed (error is the same). I don't have Visual Studio installed if that's relevant at all? Is there a parameter I can pass to pymc3 to force it to not use the compiler, even if just to check the problem is confined to compiling issues alone?
You can set the Theano compile mode to py
. That should be possible via an environment variable, but I don't remember which one.
Alternatively try theano.config.mode = theano.compile.Mode("py")
before importing PyMC3.
Imports looked like this:
import pandas as pd
import numpy as np
import theano
theano.config.mode = theano.compile.Mode("py")
import pymc3 as pm
Same error as before...
I appreciated the quick responses! Am I to understand these issues seem to be mostly confined to windows?
This problem in particular may be Windows specific, but more in the sense that there is something wrong with the compilation dependencies that arises from an unlucky combination of packages.. The dependency tree looks different by platform.. Do you depend on PyMC3 version 3, or could you already use the new v4? We're close to a release and because it's switching the backend you might get lucky. Otherwise Docker or WSL are fallback options. I know it is far from ideal, but depending on what you want to do, they might bridge you to the fun parts faster.
Can you post a link to where I can get v4? https://github.com/pymc-devs/pymc4 this url says that project has ceased?
Can you post a link to where I can get v4? https://github.com/pymc-devs/pymc4 this url says that project has ceased?
We're referring to v4
as the next release that will originate from the main
branch of https://github.com/pymc-devs/pymc3
Hi again - that seems to have solved my compilation issues. However I seem to have a problem with DensityDist where I'm getting the below error. I note that the code example in the source code also breaks with this error (if fails on normal_dist.logp needing replaced with pm.Normal.logp first but subsequently fails with the same error). Am I correct in assuming this part of the codebase is just unfinished for the new version? Would commenting out the references to cls.rv_op.ndim_supp suffice as a quick fix? If this isn't the right place to be asking this question please to point me in the right direction! Thanks again for all your time.
---> 15 ll = pm.DensityDist('ll', logp, observed={'d': dv})
16 trace = pm.sample(1500, cores=-1, step = pm.NUTS())
17 trace = trace[250:]
c:\repos\pymc3\pymc3\distributions\distribution.py in __new__(cls, name, rng, dims, initval, observed, total_size, transform, *args, **kwargs)
205 # Create the RV without specifying initval, because the initval may have a shape
206 # that only matches after replicating with a size implied by dims (see below).
--> 207 rv_out = cls.dist(*args, rng=rng, initval=None, **kwargs)
208 ndim_actual = rv_out.ndim
209 resize_shape = None
c:\repos\pymc3\pymc3\distributions\distribution.py in dist(cls, dist_params, shape, size, initval, **kwargs)
289
290 create_size, ndim_expected, ndim_batch, ndim_supp = find_size(
--> 291 shape=shape, size=size, ndim_supp=cls.rv_op.ndim_supp
292 )
293 # Create the RV with a `size` right away.
AttributeError: 'NoneType' object has no attribute 'ndim_supp'
Is this on main
? DensityDist hasn't been ported yet.
DensityDist
is one of the few distributions that are not yet refactored.
Also see #4831.
I am not familiar with DensityDist
and can't tell how complicated the refactor is.
Thanks guys this is all very helpful to know. Is there a vague estimate on when those distributions will be refactored, or when the v4 release is planned?
Hi again - that seems to have solved my compilation issues. However I seem to have a problem with DensityDist where I'm getting the below error. I note that the code example in the source code also breaks with this error (if fails on normal_dist.logp needing replaced with pm.Normal.logp first but subsequently fails with the same error). Am I correct in assuming this part of the codebase is just unfinished for the new version? Would commenting out the references to cls.rv_op.ndim_supp suffice as a quick fix? If this isn't the right place to be asking this question please to point me in the right direction! Thanks again for all your time.
---> 15 ll = pm.DensityDist('ll', logp, observed={'d': dv}) 16 trace = pm.sample(1500, cores=-1, step = pm.NUTS()) 17 trace = trace[250:] c:\repos\pymc3\pymc3\distributions\distribution.py in __new__(cls, name, rng, dims, initval, observed, total_size, transform, *args, **kwargs) 205 # Create the RV without specifying initval, because the initval may have a shape 206 # that only matches after replicating with a size implied by dims (see below). --> 207 rv_out = cls.dist(*args, rng=rng, initval=None, **kwargs) 208 ndim_actual = rv_out.ndim 209 resize_shape = None c:\repos\pymc3\pymc3\distributions\distribution.py in dist(cls, dist_params, shape, size, initval, **kwargs) 289 290 create_size, ndim_expected, ndim_batch, ndim_supp = find_size( --> 291 shape=shape, size=size, ndim_supp=cls.rv_op.ndim_supp 292 ) 293 # Create the RV with a `size` right away. AttributeError: 'NoneType' object has no attribute 'ndim_supp'
Hi, so did you solve the problem by installing the pymc4 instead of pymc3? I also met the same problem as you, and I followed up the installation guide from here, but the problem still existed.
conda install m2w64-toolchain
I followed up the same process, but I still got some errors, could you please help me solve this problem? #4922
I have the similar problem, I think maybe I have solved it. I try to find the code that compile the network in theano, I find it is in /site-packges/theano/gof/cmodule.py. I guess that the cxxflags is the parameters that will be passed to gcc, I add
cxxflags.append('-fno-asynchronous-unwind-tables')
in line 2191. Maybe you can try my way.
@gourdchen I tried to follow you what suggested. For the cmodule.py
was in a different location. When I added the cxxflags
code, that began to pick up my RTools mingw (msys-based) compiler tools. RTools uses gcc version 8.3 but is incompatible with whatever Theano-pymc
needs. Back to the drawing board for me.
@mick-phemex same installation issues as you and @michaelosthege directed me to the discussion you are having with him. So I'm trying to see if my Pymc3 issues go away if Anaconda mingw compilers are set-up correctly in the User environment variables per recommendation by @michaelosthege at Pymc3 Discourse web site.
If I find a solution, I will post it here.
Sree
@gourdchen @mick-phemex Following the User Path variables information that @michaelosthege had posted in his conversation with you @mick-phemex I was able to overcome two issues I was dealing with:
mingw
compilers from RTools installation. So this meant pymc3
worked for some models and not others.bambi
, a really nice wrapper for pymc3
, was failing even though pymc3
appeared to be working. Bambi allows non-programmers to specify Bayesian multi-level / hierarchical models using the formula specification. (as R does, bambi
replicates the ease of use of a R package called brms
, which is a wrapper for rstan
package, first package to implement the HMC-NUTS sampler developed by Matthew Hoffman and Andrew Gelman).Both of these issues had the same underlying cause: Anaconda packages were accessing the incorrect mingw
compilers. Now that is resolved and all of my models are working.
Description of your problem
By following the Windows installation guide and the getting started notebook I get the following error when calling
map_estimate = pm.find_MAP(model=basic_model)
Please provide a minimal, self-contained, and reproducible example.
Please provide the full traceback.
Please provide any additional information below.
Are there any other requirements that are missing in the windows installation guide that are missing?
Versions and main components