Installation via conda-forge leads to error: command 'gcc' failed with exit status 1

Eh2406 commented 6 years ago

Summary:

Tried to update to pystan 2.17 using conda-forge, now can't compile models.

Description:

I spun up my vm, linux mint, for learning stan. I did a conda update --all and accepted the improvements. I noticed that I did not have pystan 2.17 so I ran conda update cython pystan -c conda-forge and installed the new version. Unfortunately building any model will rase: LinkError: command 'g++' failed with exit status 1. I don't know much about c(++) so don't know how to get more detailed error messages for linker errors.

What did I do wrong? What more info could help figure it out?

Reproducible Steps:

in regr.stan

data {
  int<lower=1> N;
  int<lower=1> M;
  matrix[M, N] X;
  vector[N] y;
}

parameters {
  vector[M] beta;
  real alpha;
  real<lower=0> sigma;
}

model {
  beta ~ normal(0, 10);
  alpha ~ normal(0, 10);
  sigma ~ cauchy(0, 10);

  y ~ normal(X' * beta + alpha, sigma);
}

in test,py

from __future__ import print_function, division
import pystan
import numpy as np
import scipy
import scipy.stats as stats
from collections import OrderedDict

np.random.seed(2481)

N  =  5000
x =  stats.norm(loc=10, scale=1).rvs(N)
X = np.c_[x,x**2].T

M = 2
beta = np.array([[2.5],[-1]])
alpha = -0.275
sigma = 0.8

mu = np.dot(X.T, beta) + alpha
y = np.squeeze(stats.norm(loc=mu, scale=sigma).rvs())

data = OrderedDict({'N': N, 'M': M, 'X':X, 'y':y})

model = pystan.StanModel(file='regr.stan')
fit = model.sampling(data=data, seed=42)

in terminal run

ipython test.py

Current Output:

INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_56ca276e23b83377730a795fa0a95b5a NOW.
---------------------------------------------------------------------------
LinkError                                 Traceback (most recent call last)
/home/test/Desktop/stan_test/qr_reg_test/test.py in <module>()
     22 data = OrderedDict({'N': N, 'M': M, 'X':X, 'y':y})
     23 
---> 24 model = pystan.StanModel(file='regr.stan')
     25 fit = model.sampling(data=data, seed=42)

/home/test/anaconda2/lib/python2.7/site-packages/pystan/model.pyc in __init__(self, file, charset, model_name, model_code, stanc_ret, boost_lib, eigen_lib, verbose, obfuscate_model_name, extra_compile_args)
    311 
    312         try:
--> 313             build_extension.run()
    314         finally:
    315             if redirect_stderr:

/home/test/anaconda2/lib/python2.7/distutils/command/build_ext.pyc in run(self)
    338 
    339         # Now actually compile and link everything.
--> 340         self.build_extensions()
    341 
    342     def check_extensions_list(self, extensions):

/home/test/anaconda2/lib/python2.7/distutils/command/build_ext.pyc in build_extensions(self)
    447 
    448         for ext in self.extensions:
--> 449             self.build_extension(ext)
    450 
    451     def build_extension(self, ext):

/home/test/anaconda2/lib/python2.7/distutils/command/build_ext.pyc in build_extension(self, ext)
    529             debug=self.debug,
    530             build_temp=self.build_temp,
--> 531             target_lang=language)
    532 
    533 

/home/test/anaconda2/lib/python2.7/distutils/ccompiler.pyc in link_shared_object(self, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    689                   libraries, library_dirs, runtime_library_dirs,
    690                   export_symbols, debug,
--> 691                   extra_preargs, extra_postargs, build_temp, target_lang)
    692 
    693     def link_executable(self, objects, output_progname, output_dir=None,

/home/test/anaconda2/lib/python2.7/distutils/unixccompiler.pyc in link(self, target_desc, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    201                 self.spawn(linker + ld_args)
    202             except DistutilsExecError, msg:
--> 203                 raise LinkError, msg
    204         else:
    205             log.debug("skipping %s (up-to-date)", output_filename)

LinkError: command 'g++' failed with exit status 1

Expected Output:

Not to crash.

PyStan Version:

2.17 from conda-forge

Python Version:

Anaconda python 2.7

Operating System:

Linex mint

ahartikainen commented 6 years ago

What is your gcc version?

gcc --version

Try to run in conda env

conda create -n stanenv python=3.6 numpy cython pystan #or use 2.7

source activate stanenv

There have been problems with Anaconda root and incompatible gcc version.

Eh2406 commented 6 years ago

test@test-VirtualBox ~/Desktop/stan_test/qr_reg_test $ gcc --version
gcc (GCC) 4.8.5
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
test@test-VirtualBox ~/Desktop/stan_test/qr_reg_test $ conda create -n stanenv python=2.7 numpy cython pystan
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment /home/test/anaconda2/envs/stanenv:

The following NEW packages will be INSTALLED:

    ca-certificates: 2017.08.26-h1d4fec5_0     
    certifi:         2017.7.27.1-py27h9ceb091_0
    cython:          0.26.1-py27hdbcff32_0     
    intel-openmp:    2018.0.0-h15fc484_7       
    libedit:         3.1-heed3624_0            
    libffi:          3.2.1-h4deb6c0_3          
    libgcc-ng:       7.2.0-h7cc24e2_2          
    libstdcxx-ng:    7.2.0-h7a57d05_2          
    mkl:             2018.0.0-hb491cac_4       
    ncurses:         6.0-h06874d7_1            
    numpy:           1.13.3-py27hbcc08e0_0     
    openssl:         1.0.2l-h077ae2c_5         
    pip:             9.0.1-py27ha730c48_4      
    pystan:          2.14.0.0-py27_0           
    python:          2.7.14-h89e7a4a_22        
    readline:        7.0-hac23ff0_3            
    setuptools:      36.5.0-py27h68b189e_0     
    sqlite:          3.20.1-h6d8b0f3_1         
    tk:              8.6.7-h5979e9b_1          
    wheel:           0.29.0-py27h411dd7b_1     
    zlib:            1.2.11-hfbfcf68_1         

Proceed ([y]/n)? y

python-2.7.14- 100% |################################| Time: 0:00:04   2.76 MB/s
pip-9.0.1-py27 100% |################################| Time: 0:00:00   3.70 MB/s
#
# To activate this environment, use:
# > source activate stanenv
#
# To deactivate an active environment, use:
# > source deactivate
#

test@test-VirtualBox ~/Desktop/stan_test/qr_reg_test $ source activate stanenv
(stanenv) test@test-VirtualBox ~/Desktop/stan_test/qr_reg_test $ ipython test.py
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_56ca276e23b83377730a795fa0a95b5a NOW.
---------------------------------------------------------------------------
LinkError                                 Traceback (most recent call last)
/home/test/Desktop/stan_test/qr_reg_test/test.py in <module>()
     22 data = OrderedDict({'N': N, 'M': M, 'X':X, 'y':y})
     23 
---> 24 model = pystan.StanModel(file='regr.stan')
     25 fit = model.sampling(data=data, seed=42)

/home/test/anaconda2/lib/python2.7/site-packages/pystan/model.pyc in __init__(self, file, charset, model_name, model_code, stanc_ret, boost_lib, eigen_lib, verbose, obfuscate_model_name, extra_compile_args)
    311 
    312         try:
--> 313             build_extension.run()
    314         finally:
    315             if redirect_stderr:

/home/test/anaconda2/lib/python2.7/distutils/command/build_ext.pyc in run(self)
    338 
    339         # Now actually compile and link everything.
--> 340         self.build_extensions()
    341 
    342     def check_extensions_list(self, extensions):

/home/test/anaconda2/lib/python2.7/distutils/command/build_ext.pyc in build_extensions(self)
    447 
    448         for ext in self.extensions:
--> 449             self.build_extension(ext)
    450 
    451     def build_extension(self, ext):

/home/test/anaconda2/lib/python2.7/distutils/command/build_ext.pyc in build_extension(self, ext)
    529             debug=self.debug,
    530             build_temp=self.build_temp,
--> 531             target_lang=language)
    532 
    533 

/home/test/anaconda2/lib/python2.7/distutils/ccompiler.pyc in link_shared_object(self, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    689                   libraries, library_dirs, runtime_library_dirs,
    690                   export_symbols, debug,
--> 691                   extra_preargs, extra_postargs, build_temp, target_lang)
    692 
    693     def link_executable(self, objects, output_progname, output_dir=None,

/home/test/anaconda2/lib/python2.7/distutils/unixccompiler.pyc in link(self, target_desc, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    201                 self.spawn(linker + ld_args)
    202             except DistutilsExecError, msg:
--> 203                 raise LinkError, msg
    204         else:
    205             log.debug("skipping %s (up-to-date)", output_filename)

LinkError: command 'g++' failed with exit status 1
(stanenv) test@test-VirtualBox ~/Desktop/stan_test/qr_reg_test $ gcc --version
gcc (GCC) 4.8.5
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Is any of that helpful?

Eh2406 commented 6 years ago

Oops... Let's try that with -c conda-forge

test@test-VirtualBox ~/Desktop/stan_test/qr_reg_test $ conda create -n stanenv python=2.7 numpy cython pystan  -c conda-forge
Fetching package metadata .............
Solving package specifications: .

Package plan for installation in environment /home/test/anaconda2/envs/stanenv:

The following NEW packages will be INSTALLED:

    backports:                     1.0-py27_1                    conda-forge
    backports.functools_lru_cache: 1.4-py27_1                    conda-forge
    backports_abc:                 0.5-py27_0                    conda-forge
    blas:                          1.1-openblas                  conda-forge
    ca-certificates:               2017.7.27.1-0                 conda-forge
    certifi:                       2017.7.27.1-py27_0            conda-forge
    cycler:                        0.10.0-py27_0                 conda-forge
    cython:                        0.27.2-py27_0                 conda-forge
    dbus:                          1.10.22-0                     conda-forge
    expat:                         2.2.1-0                       conda-forge
    fontconfig:                    2.12.1-4                      conda-forge
    freetype:                      2.7-2                         conda-forge
    functools32:                   3.2.3.2-py27_1                conda-forge
    gettext:                       0.19.7-1                      conda-forge
    glib:                          2.51.4-0                      conda-forge
    gst-plugins-base:              1.8.0-0                       conda-forge
    gstreamer:                     1.8.0-2                       conda-forge
    icu:                           58.1-1                        conda-forge
    jpeg:                          9b-1                          conda-forge
    libffi:                        3.2.1-3                       conda-forge
    libgfortran:                   3.0.0-1                                  
    libiconv:                      1.14-4                        conda-forge
    libpng:                        1.6.28-2                      conda-forge
    libxcb:                        1.12-1                        conda-forge
    libxml2:                       2.9.5-1                       conda-forge
    matplotlib:                    2.1.0-py27_1                  conda-forge
    ncurses:                       5.9-10                        conda-forge
    numpy:                         1.13.3-py27_blas_openblas_200 conda-forge [blas_openblas]
    openblas:                      0.2.19-2                      conda-forge
    openssl:                       1.0.2l-0                      conda-forge
    pcre:                          8.39-0                        conda-forge
    pip:                           9.0.1-py27_0                  conda-forge
    pyparsing:                     2.2.0-py27_0                  conda-forge
    pyqt:                          5.6.0-py27_4                  conda-forge
    pystan:                        2.17.0.0-py27_0               conda-forge
    python:                        2.7.14-1                      conda-forge
    python-dateutil:               2.6.1-py27_0                  conda-forge
    pytz:                          2017.3-py_1                   conda-forge
    qt:                            5.6.2-3                       conda-forge
    readline:                      6.2-0                         conda-forge
    setuptools:                    36.6.0-py27_1                 conda-forge
    singledispatch:                3.4.0.3-py27_0                conda-forge
    sip:                           4.18-py27_1                   conda-forge
    six:                           1.11.0-py27_1                 conda-forge
    sqlite:                        3.13.0-1                      conda-forge
    ssl_match_hostname:            3.5.0.1-py27_1                conda-forge
    subprocess32:                  3.2.7-py27_0                  conda-forge
    tk:                            8.5.19-2                      conda-forge
    tornado:                       4.5.2-py27_0                  conda-forge
    wheel:                         0.30.0-py_1                   conda-forge
    xorg-libxau:                   1.0.8-3                       conda-forge
    xorg-libxdmcp:                 1.1.2-3                       conda-forge
    xz:                            5.2.3-0                       conda-forge
    zlib:                          1.2.11-0                      conda-forge

Proceed ([y]/n)? y

ca-certificate 100% |########################################################################################################################| Time: 0:00:00 658.40 kB/s
expat-2.2.1-0. 100% |########################################################################################################################| Time: 0:00:00   1.26 MB/s
gettext-0.19.7 100% |########################################################################################################################| Time: 0:00:03   1.55 MB/s
icu-58.1-1.tar 100% |########################################################################################################################| Time: 0:00:17   1.37 MB/s
jpeg-9b-1.tar. 100% |########################################################################################################################| Time: 0:00:00   1.64 MB/s
libffi-3.2.1-3 100% |########################################################################################################################| Time: 0:00:00   1.09 MB/s
libgfortran-3. 100% |########################################################################################################################| Time: 0:00:01 274.90 kB/s
libiconv-1.14- 100% |########################################################################################################################| Time: 0:00:01   1.50 MB/s
ncurses-5.9-10 100% |########################################################################################################################| Time: 0:00:00   1.31 MB/s
pcre-8.39-0.ta 100% |########################################################################################################################| Time: 0:00:00   1.04 MB/s
sqlite-3.13.0- 100% |########################################################################################################################| Time: 0:00:05 882.47 kB/s
tk-8.5.19-2.ta 100% |########################################################################################################################| Time: 0:00:01   1.09 MB/s
xorg-libxau-1. 100% |########################################################################################################################| Time: 0:00:00 152.37 kB/s
xorg-libxdmcp- 100% |########################################################################################################################| Time: 0:00:00 324.34 kB/s
xz-5.2.3-0.tar 100% |########################################################################################################################| Time: 0:00:01 493.06 kB/s
zlib-1.2.11-0. 100% |########################################################################################################################| Time: 0:00:00 859.04 kB/s
dbus-1.10.22-0 100% |########################################################################################################################| Time: 0:00:02 695.39 kB/s
glib-2.51.4-0. 100% |########################################################################################################################| Time: 0:00:07 834.66 kB/s
libpng-1.6.28- 100% |########################################################################################################################| Time: 0:00:00 754.26 kB/s
libxcb-1.12-1. 100% |########################################################################################################################| Time: 0:00:00 830.50 kB/s
libxml2-2.9.5- 100% |########################################################################################################################| Time: 0:00:06 846.69 kB/s
openblas-0.2.1 100% |########################################################################################################################| Time: 0:00:15 980.26 kB/s
openssl-1.0.2l 100% |########################################################################################################################| Time: 0:00:03   1.05 MB/s
readline-6.2-0 100% |########################################################################################################################| Time: 0:00:00   1.04 MB/s
blas-1.1-openb 100% |########################################################################################################################| Time: 0:00:00 682.43 kB/s
freetype-2.7-2 100% |########################################################################################################################| Time: 0:00:03 908.59 kB/s
gstreamer-1.8. 100% |########################################################################################################################| Time: 0:00:05 606.76 kB/s
python-2.7.14- 100% |########################################################################################################################| Time: 0:00:13   1.08 MB/s
backports-1.0- 100% |########################################################################################################################| Time: 0:00:00   3.29 MB/s
backports_abc- 100% |########################################################################################################################| Time: 0:00:00   1.12 MB/s
certifi-2017.7 100% |########################################################################################################################| Time: 0:00:00 751.82 kB/s
fontconfig-2.1 100% |########################################################################################################################| Time: 0:00:00 967.22 kB/s
functools32-3. 100% |########################################################################################################################| Time: 0:00:00 111.84 kB/s
gst-plugins-ba 100% |########################################################################################################################| Time: 0:00:04   1.02 MB/s
numpy-1.13.3-p 100% |########################################################################################################################| Time: 0:00:08   1.00 MB/s
pyparsing-2.2. 100% |########################################################################################################################| Time: 0:00:00 604.56 kB/s
pytz-2017.3-py 100% |########################################################################################################################| Time: 0:00:00 513.48 kB/s
sip-4.18-py27_ 100% |########################################################################################################################| Time: 0:00:00 736.69 kB/s
six-1.11.0-py2 100% |########################################################################################################################| Time: 0:00:00 453.62 kB/s
ssl_match_host 100% |########################################################################################################################| Time: 0:00:00   7.88 MB/s
subprocess32-3 100% |########################################################################################################################| Time: 0:00:00 526.77 kB/s
cycler-0.10.0- 100% |########################################################################################################################| Time: 0:00:00 332.54 kB/s
python-dateuti 100% |########################################################################################################################| Time: 0:00:00 428.22 kB/s
qt-5.6.2-3.tar 100% |########################################################################################################################| Time: 0:00:39   1.20 MB/s
setuptools-36. 100% |########################################################################################################################| Time: 0:00:00 909.09 kB/s
singledispatch 100% |########################################################################################################################| Time: 0:00:00   2.54 MB/s
backports.func 100% |########################################################################################################################| Time: 0:00:00   2.12 MB/s
pyqt-5.6.0-py2 100% |########################################################################################################################| Time: 0:00:04   1.29 MB/s
tornado-4.5.2- 100% |########################################################################################################################| Time: 0:00:00 823.29 kB/s
wheel-0.30.0-p 100% |########################################################################################################################| Time: 0:00:00 484.59 kB/s
matplotlib-2.1 100% |########################################################################################################################| Time: 0:00:07   1.28 MB/s
pip-9.0.1-py27 100% |########################################################################################################################| Time: 0:00:01 945.12 kB/s
#
# To activate this environment, use:
# > source activate stanenv
#
# To deactivate an active environment, use:
# > source deactivate
#
test@test-VirtualBox ~/Desktop/stan_test/qr_reg_test $ source activate stanenv
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_56ca276e23b83377730a795fa0a95b5a NOW.
---------------------------------------------------------------------------
LinkError                                 Traceback (most recent call last)
/home/test/Desktop/stan_test/qr_reg_test/test.py in <module>()
     22 data = OrderedDict({'N': N, 'M': M, 'X':X, 'y':y})
     23 
---> 24 model = pystan.StanModel(file='regr.stan')
     25 fit = model.sampling(data=data, seed=42)

/home/test/anaconda2/lib/python2.7/site-packages/pystan/model.pyc in __init__(self, file, charset, model_name, model_code, stanc_ret, boost_lib, eigen_lib, verbose, obfuscate_model_name, extra_compile_args)
    311 
    312         try:
--> 313             build_extension.run()
    314         finally:
    315             if redirect_stderr:

/home/test/anaconda2/lib/python2.7/distutils/command/build_ext.pyc in run(self)
    338 
    339         # Now actually compile and link everything.
--> 340         self.build_extensions()
    341 
    342     def check_extensions_list(self, extensions):

/home/test/anaconda2/lib/python2.7/distutils/command/build_ext.pyc in build_extensions(self)
    447 
    448         for ext in self.extensions:
--> 449             self.build_extension(ext)
    450 
    451     def build_extension(self, ext):

/home/test/anaconda2/lib/python2.7/distutils/command/build_ext.pyc in build_extension(self, ext)
    529             debug=self.debug,
    530             build_temp=self.build_temp,
--> 531             target_lang=language)
    532 
    533 

/home/test/anaconda2/lib/python2.7/distutils/ccompiler.pyc in link_shared_object(self, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    689                   libraries, library_dirs, runtime_library_dirs,
    690                   export_symbols, debug,
--> 691                   extra_preargs, extra_postargs, build_temp, target_lang)
    692 
    693     def link_executable(self, objects, output_progname, output_dir=None,

/home/test/anaconda2/lib/python2.7/distutils/unixccompiler.pyc in link(self, target_desc, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    201                 self.spawn(linker + ld_args)
    202             except DistutilsExecError, msg:
--> 203                 raise LinkError, msg
    204         else:
    205             log.debug("skipping %s (up-to-date)", output_filename)

LinkError: command 'g++' failed with exit status 1

bah humbug, same thing.

ahartikainen commented 6 years ago

Test conda gcc (its version should be 5.x, this will install only for the environment, so it's not touching system gcc)

conda install gcc

Also, have you tested that the pip version works?

pip install pystan

Eh2406 commented 6 years ago

Looks like conda is still making the transition. When did Stan start requiring 5.x? I'm just surprised as it was working on this vm befor.

(stanenv) test@test-VirtualBox ~/Desktop/stan_test/qr_reg_test $ conda update gcc -c conda-forge
Fetching package metadata .............
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /home/test/anaconda2/envs/stanenv:
#
gcc                       4.8.5                         7

same error with the pip version.

ahartikainen commented 6 years ago

It is not a problem with PyStan/Stan, it is a problem with Anaconda.

Somewhere there is probably libstdc++ file pointing to 5.x gcc.

edit. I will get back to this later (few hours) (accidentally closed the issue)

ahartikainen commented 6 years ago

Hi, thanks for your patience, this could take a few iterations, but it is a problem we can solve.

In the stanenv do:

conda install gcc libgcc

This could fix the problem.

edit. The linux mint is probably using gcc 5.x as is the ubuntu. The gcc with Anaconda was 4.x, but recently it was updated. You may need to update the anaconda distribution. In the root (conda root / the default conda environment when you open your terminal, not sudo) environment.

conda update conda
conda update anaconda

Eh2406 commented 6 years ago

Thanks for the edits. I updated conda and anaconda but still gcc 4.x. which gcc confirms that it comes from anaconda.

I'm wondering if I should just reinstall anaconda.

Eh2406 commented 6 years ago

Uninstalling and reinstalling anaconda worked! Thank you for all the help.

Perhaps we can make a clearer error message for when gcc is not up to snuff?

ahartikainen commented 6 years ago

Great that everything works now.

We probably need to update the instructions considering Anaconda distribution. This will happen from time to time when they update their distribution and it doesn't update the old ones correctly.

ps. Just a reminder. On linux the recommended way to install gcc/g++ is to use.

apt install build-essential

mozzhorin commented 6 years ago

I have the same problem. I've used an instance from AWS with Ubuntu 16.04 (14.04 I've tried too), Anaconda3 5.0.1 or 4.2.0 or the latest Miniconda3. I've tried all solutions from this topic and from #384 and #378. I've even tried to uninstall and install again Miniconda. Could you please give new instructions to installation PyStan on Linux so it can work? Because the old one doesn't. Thank you.

ahartikainen commented 6 years ago

Hi, can you also give your gcc version and where it is

gcc --version

which gcc

When you uninstalled Miniconda, you didn't have any other Anaconda distribution installed on the same machine?

mozzhorin commented 6 years ago

(cstan) ubuntu@ip-172-31-24-66:~$ gcc --version gcc (Ubuntu 5.4.0-6ubuntu1~16.04.5) 5.4.0 20160609 ... (cstan) ubuntu@ip-172-31-24-66:~$ which gcc /home/ubuntu/miniconda3/envs/cstan/bin/gcc

But when I start Python, it says: (cstan) ubuntu@ip-172-31-24-66:~$ python Python 3.6.3 | packaged by conda-forge | (default, Nov 4 2017, 10:10:56) [GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux

When I uninstalled Miniconda, there were no other Anaconda distribution on this machine. I've done it one more time from a clean Ubuntu image, the hole log is in the file. aws-ubuntu-miniconda-pystan-log.txt

ahartikainen commented 6 years ago

This is a kind of weird problem. It is probably something that will go away when gcc 5.x becomes default for all.

I guess it is related to this issue: https://github.com/conda-forge/staged-recipes/issues/2472

So here are the first steps to solve this.

All the gcc things should be done with apt-get but are here done with conda excluding the first test.

Install fresh Miniconda
apt-get install build-essentials
conda update conda

0st scenario:

create a new environment (without install gcc from conda, use system gcc)

conda create -n stan_env0 python=3.6 source activate stan_env0 pip install pystan # gcc ? (5.1.3?)

1st scenario:

create a new environment (without install gcc from conda, use system gcc)

conda create -n stan_env1 python=3.6 source activate stan_env1 conda install -c conda-forge numpy cython matplotlib scipy pystan # gcc ? (5.1.3?)

2st scenario:

create a new environment

conda create -n stan_env2 python=3.6 source activate stan_env2 conda install -c conda-forge numpy cython matplotlib scipy pystan conda install -c anaconda gcc libgcc # gcc 4.8.5

3rd scenario

create a new environment

conda create -n stan_env3 python=3.6 source activate stan_env3 conda install -c conda-forge numpy cython matplotlib scipy pystan conda install -c salford_systems gcc-5 libgcc-5 # gcc 5.4.0

4th scenario

create a new environment

conda create -n stan_env3 python=3.6 source activate stan_env3 conda install -c conda-forge numpy cython matplotlib scipy pystan conda install -c quantstack gcc-6 libgcc-6 # gcc 6.1.0

I will have to test this a bit on Docker when I have some time.

mozzhorin commented 6 years ago

So I've tried all five scenarios, no one from them works. The command "gcc --version" shows every time the version, that I've just installed, but python shows every time (except for 0th scenario), that it uses GCC 4.4.7. Only in the 0th scenario it uses GCC 7.2.0, but it steel doesn't compile. The full log is in the file. aws-ubuntu-miniconda-5scenarios-log.txt

ahartikainen commented 6 years ago

You mean this?

[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]

This is just the compiler used to compile the Python. Not sure if it's related to our problem.

mozzhorin commented 6 years ago

Yes, I meant this. If it doesn't matter, than ok.

CarstVaartjes commented 6 years ago

I had a similar issue in amazon_linux and solved it by explicitly removing gcc with yum and then

 yum install gcc64
 yum install gcc64-c++

It looks like that solved it

ahartikainen commented 6 years ago

Weird (or not really). Update:

I tested ubuntu:latest (16.04) on Docker.

This worked

apt install build-essential python3 python3-dev python3-pip
python3 -m pip install pystan

also getting latest miniconda

conda create -n stan python=3.6 pystan

worked. Now I remember that it is an old PyStan, but still no error.

Have to still test with the latest conda-forge version.

mozzhorin commented 6 years ago

@ahartikainen That doesn't work by me. Can it be a problem with the AWS Ubuntu image? So one more time: I launch a new AWS instance with a clean Ubuntu 16.04. sudo apt install... couldn't find python3-pip, so I make first sudo apt-get update and then
sudo apt-get install build-essential python3 python3-dev python3-pip python3 -m pip install pystan it installs PyStan 2.17.0. Then I install the latest Miniconda

wget -c http://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b -p ~/miniconda3
export PATH=~/miniconda3/bin:$PATH

conda create -n stan python=3.6 pystan
source activate stan
python

and then I try a simple example from this page https://pystan.readthedocs.io/en/latest/windows.html

>>> import pystan
>>> model_code = 'parameters {real y;} model {y ~ normal(0,1);}'
>>> model = pystan.StanModel(model_code=model_code)

here I get the error command '/home/ubuntu/miniconda3/envs/stan/bin/x86_64-conda_cos6-linux-gnu-cc' failed with exit status 1 The full log is in the file. aws-ubuntu-log-18-11-17.txt

mozzhorin commented 6 years ago

@CarstVaartjes Could you please give all your steps? Because there is definitely some magic involved.

ahartikainen commented 6 years ago

Ok, I went and did AWS + ubuntu. I may have found the reason for the error.

Try running your model compilation with verbose=True

sm = pystan.StanModel(model_code=model_code, verbose=True)

There will be a lot of warnings. Ignore these.

Then the real error is not obfuscated by the cython

virtual memory exhausted: Cannot allocate memory

That error is probably from OS/C++ so Cython is hiding it from the view.

ahartikainen commented 6 years ago

One option is to compile models in local system (docker) and pickle the compiled model. Then sample from the unpickled model. This assumes that the model sampling doesn't take too much space. Otherwise CmdStan should be used for low-memory sampling.

mozzhorin commented 6 years ago

@ahartikainen Yes, that was a memory problem. I've tried the code on another AWS instance with more memory, and it works. Thank you for your help!

sbitzer commented 6 years ago

Sorry to open this up again, but I'm experiencing some strange pystan-compile errors that may be memory-related and I wonder whether I can get around them.

The problem is that I have a model that I can compile in an ipython interpreter, but only if I do not invoke an interactive gui, e.g., with matplotlib qt5 before compilation. The model below does not compile with qt5 invoked, but it does, if I simply remove the - 0.5 from the definition of mu in the transformed parameters block.

The error that I get is non-informative:

CompileError: command '/home/bitzer/programs/anaconda/envs/stan/bin/x86_64-conda_cos6-linux-gnu-cc' failed with exit status 1

The verbose compile messages stop at

/lib/stan_math/lib/eigen_3.3.3/Eigen/src/Core/AssignEvaluator.h:90:50: warning: enum constant in boolean context [-Wint-in-bool-context]
     MaySliceVectorize  = bool(MightVectorize) && bool(DstHasDirectAccess)
                                                  ^~~~~~~~~~~~~~~~~~~~~~~~

and I couldn't find a message along the lines virtual memory exhausted. Also only around 50% of the memory of my system is used during compilation. So if this is a memory issue, I suspect that I can somewhere increase a setting for the allowed virtual memory used during compilation, but where?

model used

data {
    int<lower=1> N;
    vector[N] x;
    vector[N] y;
}

parameters {
    real beta;
    real c;
    real<lower=0> sigma;
}

transformed parameters {
    vector[N] mu = beta * 2 * (inv_logit(x) - 0.5) + c;
}

model {
    beta ~ normal(0, 1);
    c ~ normal(0, 1);
    sigma ~ exponential(1);

    y ~ normal(mu, sigma);
}

version infos

I tested all this under Mint 17.3 (Ubuntu 14.04) in a conda environment created with:

conda create -n stan python=3.5 pandas seaborn ipython spyder cython pystan numpy scipy

The pystan version is 2.17.0.0-py35h541c666_0 and it uses GCC 7.2.0 installed by conda for compiling models.

ahartikainen commented 6 years ago

Can you paste the verbose output? There should be one line with an error message. It does not need to be the last line.

sbitzer commented 6 years ago

I just tried to do that, but the output has too many lines for my console for simple copy & paste. So I started IPython with

ipython 2> stan.log

in IPython I then run

matplotlib qt5
run test_stan.py

Compiling works without errors and compiler warnings are saved in stan.log. On the other hand, if I start IPython just with

ipython

and enter the same in the interpreter as above, I get the error, but can't capture all verbose output. Any ideas how to get around that?

Auerilas commented 6 years ago

I was hoping to follow up on this with a weird issue that is partially related to this issue that arose as I was installing pystan, but is more of an Anaconda issue (running Linux Mint 18.3). I recently reinstalled Anaconda. Then, I did

conda install pystan

That gave me a gcc compile error. To fix this, I did:

conda install gcc libgcc

which worked fine. Oddly, conda and conda-forge only have gcc 4.8.7 available, whereas the default version on my machine is 5.4.0. Unfortunately, I had some other, unrelated issues with scipy. I resolved those with

apt-get install g++

which put the newest, 5.4.0 version of g++ on my machine. That fixed scipy, as well. All was good. However, I compared my install on this machine to my laptop with the same setup. On my desktop,

gcc -v

gave me the Anaconda 4.8 version, and on my laptop it gave me the Ubuntu 5.4 version. I also noticed that I didn't have conda gcc installed on my laptop, just libgcc. So, I uninstalled gcc from my current machine. Now, I get

gcc -v
-bash: /home/nate/anaconda3/bin/gcc: No such file or directory

which gcc
/usr/bin/gcc

g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.9' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9)

Oddly, both scipy and pystan and running fine! But I can't use gcc from the command line. Even though the which command is pointing to a binary, Anaconda seems to override it. Any thoughts?

UPDATE Ugh. Nevermind. A reboot fixed it. This all started by trying to revert to MLK numpy. After hours of installing and reinstalling and breaking everything, all I had to do was install g++....

stan-dev / pystan2