search

stan-dev / pystan2

PyStan, the Python interface to Stan
GNU General Public License v3.0
920 stars 189 forks source link

LinkError: Command "x86_64-apple-darwin13.4.0-clang++ ..." failed with exit status 1 #730

Closed jankaWIS closed 4 years ago

jankaWIS commented 4 years ago

Summary:

I'm not sure whether this is the correct place to ask this question, if not please redirect me (I feel like the STAN forum points here in this case). I am compiling a model in pystan and I am getting the following problem which I have no idea what means and what is wrong. My guess would be some troubles with compilers since I have previously encountered another issue with clang but I am shooting in dark. Has anyone seen this before or knows what the problem is?

Description:

LinkError: Command "x86_64-apple-darwin13.4.0-clang++ -bundle -undefined dynamic_lookup -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/jan/anaconda3/lib -L/Users/jan/anaconda3/lib -flto -Wl,-export_dynamic -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/jan/anaconda3/lib -L/Users/jan/anaconda3/lib -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/jan/anaconda3/lib -L/Users/jan/anaconda3/lib -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem /Users/jan/anaconda3/include -D_FORTIFY_SOURCE=2 -mmacosx-version-min=10.9 -isystem /Users/jan/anaconda3/include -arch x86_64 /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_dg2mqj8q/var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_dg2mqj8q/stanfit4anon_model_649148968e47447c8e6aa386bb89c275_8781054055478004166.o -o /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_dg2mqj8q/stanfit4anon_model_649148968e47447c8e6aa386bb89c275_8781054055478004166.cpython-38-darwin.so" failed with exit status 1

The entire message goes as follows:

INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_649148968e47447c8e6aa386bb89c275 NOW.

---------------------------------------------------------------------------
DistutilsExecError                        Traceback (most recent call last)
~/anaconda3/lib/python3.8/distutils/unixccompiler.py in link(self, target_desc, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    204 
--> 205                 self.spawn(linker + ld_args)
    206             except DistutilsExecError as msg:

~/anaconda3/lib/python3.8/site-packages/numpy/distutils/ccompiler.py in <lambda>(self, *args, **kw)
     89     # Py3k does not have unbound method anymore, MethodType does not work
---> 90     m = lambda self, *args, **kw: func(self, *args, **kw)
     91     setattr(klass, method_name, m)

~/anaconda3/lib/python3.8/site-packages/numpy/distutils/ccompiler.py in CCompiler_spawn(self, cmd, display)
    172         msg = ''
--> 173     raise DistutilsExecError('Command "%s" failed with exit status %d%s' %
    174                             (cmd, s, msg))

DistutilsExecError: Command "x86_64-apple-darwin13.4.0-clang++ -bundle -undefined dynamic_lookup -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/jan/anaconda3/lib -L/Users/jan/anaconda3/lib -flto -Wl,-export_dynamic -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/jan/anaconda3/lib -L/Users/jan/anaconda3/lib -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/jan/anaconda3/lib -L/Users/jan/anaconda3/lib -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem /Users/jan/anaconda3/include -D_FORTIFY_SOURCE=2 -mmacosx-version-min=10.9 -isystem /Users/jan/anaconda3/include -arch x86_64 /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_dg2mqj8q/var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_dg2mqj8q/stanfit4anon_model_649148968e47447c8e6aa386bb89c275_8781054055478004166.o -o /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_dg2mqj8q/stanfit4anon_model_649148968e47447c8e6aa386bb89c275_8781054055478004166.cpython-38-darwin.so" failed with exit status 1

During handling of the above exception, another exception occurred:

LinkError                                 Traceback (most recent call last)
<ipython-input-101-49011489536c> in <module>
      1 # Compile the model
      2 start_compile = time.time()
----> 3 stan_model_noLA = pystan.StanModel(model_code=model_simple_exponential_test)
      4 end_compile = time.time()
      5 print('Duration of compilation: ', timedelta(seconds=end_compile - start_compile))

~/anaconda3/lib/python3.8/site-packages/pystan/model.py in __init__(self, file, charset, model_name, model_code, stanc_ret, include_paths, boost_lib, eigen_lib, verbose, obfuscate_model_name, extra_compile_args, allow_undefined, include_dirs, includes)
    376 
    377         try:
--> 378             build_extension.run()
    379         finally:
    380             if redirect_stderr:

~/anaconda3/lib/python3.8/distutils/command/build_ext.py in run(self)
    338 
    339         # Now actually compile and link everything.
--> 340         self.build_extensions()
    341 
    342     def check_extensions_list(self, extensions):

~/anaconda3/lib/python3.8/distutils/command/build_ext.py in build_extensions(self)
    447             self._build_extensions_parallel()
    448         else:
--> 449             self._build_extensions_serial()
    450 
    451     def _build_extensions_parallel(self):

~/anaconda3/lib/python3.8/distutils/command/build_ext.py in _build_extensions_serial(self)
    472         for ext in self.extensions:
    473             with self._filter_build_errors(ext):
--> 474                 self.build_extension(ext)
    475 
    476     @contextlib.contextmanager

~/anaconda3/lib/python3.8/distutils/command/build_ext.py in build_extension(self, ext)
    548         language = ext.language or self.compiler.detect_language(sources)
    549 
--> 550         self.compiler.link_shared_object(
    551             objects, ext_path,
    552             libraries=self.get_libraries(ext),

~/anaconda3/lib/python3.8/distutils/ccompiler.py in link_shared_object(self, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    711                            build_temp=None,
    712                            target_lang=None):
--> 713         self.link(CCompiler.SHARED_OBJECT, objects,
    714                   output_filename, output_dir,
    715                   libraries, library_dirs, runtime_library_dirs,

~/anaconda3/lib/python3.8/distutils/unixccompiler.py in link(self, target_desc, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    205                 self.spawn(linker + ld_args)
    206             except DistutilsExecError as msg:
--> 207                 raise LinkError(msg)
    208         else:
    209             log.debug("skipping %s (up-to-date)", output_filename)

LinkError: Command "x86_64-apple-darwin13.4.0-clang++ -bundle -undefined dynamic_lookup -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/jan/anaconda3/lib -L/Users/jan/anaconda3/lib -flto -Wl,-export_dynamic -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/jan/anaconda3/lib -L/Users/jan/anaconda3/lib -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/jan/anaconda3/lib -L/Users/jan/anaconda3/lib -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem /Users/jan/anaconda3/include -D_FORTIFY_SOURCE=2 -mmacosx-version-min=10.9 -isystem /Users/jan/anaconda3/include -arch x86_64 /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_dg2mqj8q/var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_dg2mqj8q/stanfit4anon_model_649148968e47447c8e6aa386bb89c275_8781054055478004166.o -o /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_dg2mqj8q/stanfit4anon_model_649148968e47447c8e6aa386bb89c275_8781054055478004166.cpython-38-darwin.so" failed with exit status 1

Reproducible Steps:

I believe this should be all to be able to run it: Set up:

import pystan
import pickle

import numpy as np
import pandas as pd

import time
from datetime import timedelta

a = 5
k1 = 1.3
x_test = np.linspace(1,10,20)
y_test = a*np.exp(-k1*x_test)

# Data for Stan
data_stan_test = {'N' : 1, 
             'x' : x_test, 
             'y': y_test}

Stan code:

model_simple_exponential_test = """
data {
    int<lower=1> N;           // number of data points, probably not needed but helps
    vector<lower=0>[N] x;           //[N, T]; in case with T
    vector<lower=0>[N] y;      //[N, T]; in case with T
}

parameters {
  real c;
  real k1;
  real<lower=0> sigma;
  //vector[N] m;
}

transformed parameters {
  vector[N] m;
  m = c* exp(k1 * x);
}

model {
  // priors
    c ~ normal( 10 , 3 );
    k1 ~ lognormal( 0.5 , 0.2 );   
    sigma ~ cauchy(0,5); 

  // likelihood
    y ~ normal(m, sigma);
}

generated quantities {
  vector[N] ypred;
  vector[N] log_lik;

  for (sample in 1:N) {
    ypred[sample] = normal_rng(c* exp(k1 * x[sample]), sigma);
    log_lik[sample] = normal_lpdf(y[sample] | c* exp(k1 * x[sample]), sigma);
  }

}
"""

Run:

# Compile the model
start_compile = time.time()
stan_model_noLA = pystan.StanModel(model_code=model_simple_exponential_test)
end_compile = time.time()
print('Duration of compilation: ', timedelta(seconds=end_compile - start_compile))

Watermark

pystan 2.19.1.1 arviz 0.10.0 pymc3 3.9.3 numpy 1.19.2 pandas 1.1.3 last updated: Sat Oct 31 2020

CPython 3.8.5 IPython 7.18.1 python 3.8.5 watermark 2.0.2

OS Catalina 10.15.7

ahartikainen commented 4 years ago

Hi, thanks for the issue.

Can you add verbose=True to compilation step and see if there are C++ errors?

jankaWIS commented 4 years ago

Hi @ahartikainen , I will post all the logs because I do not think there is:

Compiling /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_05o10urq/stanfit4anon_model_649148968e47447c8e6aa386bb89c275_3856974590934791165.pyx because it changed.
[1/1] Cythonizing /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_05o10urq/stanfit4anon_model_649148968e47447c8e6aa386bb89c275_3856974590934791165.pyx
building 'stanfit4anon_model_649148968e47447c8e6aa386bb89c275_3856974590934791165' extension
creating /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_05o10urq/var
creating /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_05o10urq/var/folders
creating /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_05o10urq/var/folders/bx
creating /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_05o10urq/var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp
creating /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_05o10urq/var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T
creating /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_05o10urq/var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_05o10urq
x86_64-apple-darwin13.4.0-clang -fno-strict-aliasing -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O3 -Wall -Wstrict-prototypes -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O3 -pipe -fdebug-prefix-map=${SRC_DIR}=/usr/local/src/conda/${PKG_NAME}-${PKG_VERSION} -fdebug-prefix-map=/Users/jan/anaconda3=/usr/local/src/conda-prefix -flto -Wl,-export_dynamic -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O3 -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem /Users/jan/anaconda3/include -D_FORTIFY_SOURCE=2 -mmacosx-version-min=10.9 -isystem /Users/jan/anaconda3/include -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -I/var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_05o10urq -I/Users/jan/anaconda3/lib/python3.8/site-packages/pystan -I/Users/jan/anaconda3/lib/python3.8/site-packages/pystan/stan/src -I/Users/jan/anaconda3/lib/python3.8/site-packages/pystan/stan/lib/stan_math -I/Users/jan/anaconda3/lib/python3.8/site-packages/pystan/stan/lib/stan_math/lib/eigen_3.3.3 -I/Users/jan/anaconda3/lib/python3.8/site-packages/pystan/stan/lib/stan_math/lib/boost_1.69.0 -I/Users/jan/anaconda3/lib/python3.8/site-packages/pystan/stan/lib/stan_math/lib/sundials_4.1.0/include -I/Users/jan/anaconda3/lib/python3.8/site-packages/numpy/core/include -I/Users/jan/anaconda3/include/python3.8 -c /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_05o10urq/stanfit4anon_model_649148968e47447c8e6aa386bb89c275_3856974590934791165.cpp -o /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_05o10urq/var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_05o10urq/stanfit4anon_model_649148968e47447c8e6aa386bb89c275_3856974590934791165.o -O2 -ftemplate-depth-256 -Wno-unused-function -Wno-uninitialized -std=c++1y
x86_64-apple-darwin13.4.0-clang++ -bundle -undefined dynamic_lookup -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/jan/anaconda3/lib -L/Users/jan/anaconda3/lib -flto -Wl,-export_dynamic -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/jan/anaconda3/lib -L/Users/jan/anaconda3/lib -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/jan/anaconda3/lib -L/Users/jan/anaconda3/lib -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem /Users/jan/anaconda3/include -D_FORTIFY_SOURCE=2 -mmacosx-version-min=10.9 -isystem /Users/jan/anaconda3/include -arch x86_64 /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_05o10urq/var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_05o10urq/stanfit4anon_model_649148968e47447c8e6aa386bb89c275_3856974590934791165.o -o /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_05o10urq/stanfit4anon_model_649148968e47447c8e6aa386bb89c275_3856974590934791165.cpython-38-darwin.so

---------------------------------------------------------------------------
DistutilsExecError                        Traceback (most recent call last)
~/anaconda3/lib/python3.8/distutils/unixccompiler.py in link(self, target_desc, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    204 
--> 205                 self.spawn(linker + ld_args)
    206             except DistutilsExecError as msg:

~/anaconda3/lib/python3.8/distutils/ccompiler.py in spawn(self, cmd)
    909     def spawn(self, cmd):
--> 910         spawn(cmd, dry_run=self.dry_run)
    911 

~/anaconda3/lib/python3.8/distutils/spawn.py in spawn(cmd, search_path, verbose, dry_run)
     35     if os.name == 'posix':
---> 36         _spawn_posix(cmd, search_path, dry_run=dry_run)
     37     elif os.name == 'nt':

~/anaconda3/lib/python3.8/distutils/spawn.py in _spawn_posix(cmd, search_path, verbose, dry_run)
    156                         cmd = executable
--> 157                     raise DistutilsExecError(
    158                           "command %r failed with exit status %d"

DistutilsExecError: command 'x86_64-apple-darwin13.4.0-clang++' failed with exit status 1

During handling of the above exception, another exception occurred:

LinkError                                 Traceback (most recent call last)
<ipython-input-7-f766ece53857> in <module>
      1 # Compile the model
      2 start_compile = time.time()
----> 3 stan_model_noLA = pystan.StanModel(model_code=model_simple_exponential_test, verbose=True)
      4 end_compile = time.time()
      5 print('Duration of compilation: ', timedelta(seconds=end_compile - start_compile))

~/anaconda3/lib/python3.8/site-packages/pystan/model.py in __init__(self, file, charset, model_name, model_code, stanc_ret, include_paths, boost_lib, eigen_lib, verbose, obfuscate_model_name, extra_compile_args, allow_undefined, include_dirs, includes)
    376 
    377         try:
--> 378             build_extension.run()
    379         finally:
    380             if redirect_stderr:

~/anaconda3/lib/python3.8/distutils/command/build_ext.py in run(self)
    338 
    339         # Now actually compile and link everything.
--> 340         self.build_extensions()
    341 
    342     def check_extensions_list(self, extensions):

~/anaconda3/lib/python3.8/distutils/command/build_ext.py in build_extensions(self)
    447             self._build_extensions_parallel()
    448         else:
--> 449             self._build_extensions_serial()
    450 
    451     def _build_extensions_parallel(self):

~/anaconda3/lib/python3.8/distutils/command/build_ext.py in _build_extensions_serial(self)
    472         for ext in self.extensions:
    473             with self._filter_build_errors(ext):
--> 474                 self.build_extension(ext)
    475 
    476     @contextlib.contextmanager

~/anaconda3/lib/python3.8/distutils/command/build_ext.py in build_extension(self, ext)
    548         language = ext.language or self.compiler.detect_language(sources)
    549 
--> 550         self.compiler.link_shared_object(
    551             objects, ext_path,
    552             libraries=self.get_libraries(ext),

~/anaconda3/lib/python3.8/distutils/ccompiler.py in link_shared_object(self, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    711                            build_temp=None,
    712                            target_lang=None):
--> 713         self.link(CCompiler.SHARED_OBJECT, objects,
    714                   output_filename, output_dir,
    715                   libraries, library_dirs, runtime_library_dirs,

~/anaconda3/lib/python3.8/distutils/unixccompiler.py in link(self, target_desc, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    205                 self.spawn(linker + ld_args)
    206             except DistutilsExecError as msg:
--> 207                 raise LinkError(msg)
    208         else:
    209             log.debug("skipping %s (up-to-date)", output_filename)

LinkError: command 'x86_64-apple-darwin13.4.0-clang++' failed with exit status 1
ahartikainen commented 4 years ago

Hi, do you run this on IDE (e.g. jupyter)?

Try to run your code like python filename.py

jankaWIS commented 4 years ago

Yes, I am running it on Jupyter. I will try, although that I have never tried.

ahartikainen commented 4 years ago

You can also access the C++ messages in terminal (if you have open jupyter with jupyter notebook command)

jankaWIS commented 4 years ago

That give me many, many lines. It begins with:

clang-10: warning: -Wl,-export_dynamic: 'linker' input unused [-Wunused-command-line-argument]
In file included from /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_05o10urq/stanfit4anon_model_649148968e47447c8e6aa386bb89c275_3856974590934791165.cpp:696:
In file included from /Users/jan/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4:
In file included from /Users/jan/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12:
In file included from /Users/jan/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822:
/Users/jan/anaconda3/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: 
      "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
#warning "Using deprecated NumPy API, disable it with " \

It ends with:

165 warnings generated.
ld: warning: -pie being ignored. It is only used when linking a main executable
ld: unsupported tapi file type '!tapi-tbd' in YAML file '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/libSystem.tbd' for architecture x86_64
clang-10: error: linker command failed with exit code 1 (use -v to see invocation)
ahartikainen commented 4 years ago

Hi, can you try this

https://stackoverflow.com/questions/63592445/ld-unsupported-tapi-file-type-tapi-tbd-in-yaml-file

jankaWIS commented 4 years ago

So I have tried it and it is giving me a new error:

Compiling /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_2jyq_jrw/stanfit4anon_model_649148968e47447c8e6aa386bb89c275_7455707239522855451.pyx because it changed.
[1/1] Cythonizing /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_2jyq_jrw/stanfit4anon_model_649148968e47447c8e6aa386bb89c275_7455707239522855451.pyx
building 'stanfit4anon_model_649148968e47447c8e6aa386bb89c275_7455707239522855451' extension
creating /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_2jyq_jrw/var
creating /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_2jyq_jrw/var/folders
creating /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_2jyq_jrw/var/folders/bx
creating /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_2jyq_jrw/var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp
creating /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_2jyq_jrw/var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T
creating /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_2jyq_jrw/var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_2jyq_jrw
x86_64-apple-darwin13.4.0-clang -fno-strict-aliasing -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O3 -Wall -Wstrict-prototypes -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O3 -pipe -fdebug-prefix-map=${SRC_DIR}=/usr/local/src/conda/${PKG_NAME}-${PKG_VERSION} -fdebug-prefix-map=/Users/jan/anaconda3=/usr/local/src/conda-prefix -flto -Wl,-export_dynamic -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O3 -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem /Users/jan/anaconda3/include -D_FORTIFY_SOURCE=2 -mmacosx-version-min=10.9 -isystem /Users/jan/anaconda3/include -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -I/var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_2jyq_jrw -I/Users/jan/anaconda3/lib/python3.8/site-packages/pystan -I/Users/jan/anaconda3/lib/python3.8/site-packages/pystan/stan/src -I/Users/jan/anaconda3/lib/python3.8/site-packages/pystan/stan/lib/stan_math -I/Users/jan/anaconda3/lib/python3.8/site-packages/pystan/stan/lib/stan_math/lib/eigen_3.3.3 -I/Users/jan/anaconda3/lib/python3.8/site-packages/pystan/stan/lib/stan_math/lib/boost_1.69.0 -I/Users/jan/anaconda3/lib/python3.8/site-packages/pystan/stan/lib/stan_math/lib/sundials_4.1.0/include -I/Users/jan/anaconda3/lib/python3.8/site-packages/numpy/core/include -I/Users/jan/anaconda3/include/python3.8 -c /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_2jyq_jrw/stanfit4anon_model_649148968e47447c8e6aa386bb89c275_7455707239522855451.cpp -o /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_2jyq_jrw/var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_2jyq_jrw/stanfit4anon_model_649148968e47447c8e6aa386bb89c275_7455707239522855451.o -O2 -ftemplate-depth-256 -Wno-unused-function -Wno-uninitialized -std=c++1y

---------------------------------------------------------------------------
DistutilsExecError                        Traceback (most recent call last)
~/anaconda3/lib/python3.8/distutils/unixccompiler.py in _compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts)
    116         try:
--> 117             self.spawn(compiler_so + cc_args + [src, '-o', obj] +
    118                        extra_postargs)

~/anaconda3/lib/python3.8/distutils/ccompiler.py in spawn(self, cmd)
    909     def spawn(self, cmd):
--> 910         spawn(cmd, dry_run=self.dry_run)
    911 

~/anaconda3/lib/python3.8/distutils/spawn.py in spawn(cmd, search_path, verbose, dry_run)
     35     if os.name == 'posix':
---> 36         _spawn_posix(cmd, search_path, dry_run=dry_run)
     37     elif os.name == 'nt':

~/anaconda3/lib/python3.8/distutils/spawn.py in _spawn_posix(cmd, search_path, verbose, dry_run)
    156                         cmd = executable
--> 157                     raise DistutilsExecError(
    158                           "command %r failed with exit status %d"

DistutilsExecError: command 'x86_64-apple-darwin13.4.0-clang' failed with exit status 254

During handling of the above exception, another exception occurred:

CompileError                              Traceback (most recent call last)
<ipython-input-6-f766ece53857> in <module>
      1 # Compile the model
      2 start_compile = time.time()
----> 3 stan_model_noLA = pystan.StanModel(model_code=model_simple_exponential_test, verbose=True)
      4 end_compile = time.time()
      5 print('Duration of compilation: ', timedelta(seconds=end_compile - start_compile))

~/anaconda3/lib/python3.8/site-packages/pystan/model.py in __init__(self, file, charset, model_name, model_code, stanc_ret, include_paths, boost_lib, eigen_lib, verbose, obfuscate_model_name, extra_compile_args, allow_undefined, include_dirs, includes)
    376 
    377         try:
--> 378             build_extension.run()
    379         finally:
    380             if redirect_stderr:

~/anaconda3/lib/python3.8/distutils/command/build_ext.py in run(self)
    338 
    339         # Now actually compile and link everything.
--> 340         self.build_extensions()
    341 
    342     def check_extensions_list(self, extensions):

~/anaconda3/lib/python3.8/distutils/command/build_ext.py in build_extensions(self)
    447             self._build_extensions_parallel()
    448         else:
--> 449             self._build_extensions_serial()
    450 
    451     def _build_extensions_parallel(self):

~/anaconda3/lib/python3.8/distutils/command/build_ext.py in _build_extensions_serial(self)
    472         for ext in self.extensions:
    473             with self._filter_build_errors(ext):
--> 474                 self.build_extension(ext)
    475 
    476     @contextlib.contextmanager

~/anaconda3/lib/python3.8/distutils/command/build_ext.py in build_extension(self, ext)
    526             macros.append((undef,))
    527 
--> 528         objects = self.compiler.compile(sources,
    529                                          output_dir=self.build_temp,
    530                                          macros=macros,

~/anaconda3/lib/python3.8/distutils/ccompiler.py in compile(self, sources, output_dir, macros, include_dirs, debug, extra_preargs, extra_postargs, depends)
    572             except KeyError:
    573                 continue
--> 574             self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
    575 
    576         # Return *all* object filenames, not just the ones we just built.

~/anaconda3/lib/python3.8/distutils/unixccompiler.py in _compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts)
    118                        extra_postargs)
    119         except DistutilsExecError as msg:
--> 120             raise CompileError(msg)
    121 
    122     def create_static_lib(self, objects, output_libname,

CompileError: command 'x86_64-apple-darwin13.4.0-clang' failed with exit status 254

Also, what I get in the terminal is different:

clang-10: warning: -Wl,-export_dynamic: 'linker' input unused [-Wunused-command-line-argument]
In file included from /var/folders/bx/tb4883l53hdd3zp2y0nyy_4m0000gp/T/pystan_2jyq_jrw/stanfit4anon_model_649148968e47447c8e6aa386bb89c275_7455707239522855451.cpp:57:
In file included from /Users/jan/anaconda3/include/python3.8/Python.h:25:
/Users/jan/anaconda3/bin/../include/c++/v1/stdio.h:107:15: fatal error: 'stdio.h' file not found
#include_next <stdio.h>
              ^~~~~~~~~
Stack dump:
0.  Program arguments: x86_64-apple-darwin13.4.0-clang
.
.
.
clang-10: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 10.0.0 
Target: x86_64-apple-darwin13.4.0
Thread model: posix
InstalledDir: /Users/jan/anaconda3/bin
clang-10: note: diagnostic msg: PLEASE submit a bug report to  and include the crash backtrace, preprocessed source, and associated run script.
clang-10: note: diagnostic msg: Error generating preprocessed source(s).

Btw. I tried on Windows 10 and compiled just fine.

ahartikainen commented 4 years ago

Ok, I think I know what is the problem. And I can now tell you that I don't have a real solution for it.

You have probably updated your macOS with the "upgrade" version (not a clean install) and this has left some compiler files somewhere in your path. Before this was fine, but now, the new macOS uses a different linker library to link your compiled files (old: stdlib++ --> new: libc++) and this causes so much pain.

I have not seen or tested that kind of mac, so I don't know what should be done for it to work correctly. I know clean install will fix it, but of course, that is not optimal for users. We really tried to fix that at some point, but after going deep in the system files, we couldn't figure it out, so we finally gave up and found another solution.

This solution is to use compilers from conda. You already use anaconda, so there are not too many steps to set up it.

https://pystan.readthedocs.io/en/latest/installation_beginner.html

Follow the Setting up C++ compiler

You use base anaconda, so you can use the anaconda solution. If you would switch to conda-forge environment, then I would recommend using conda-forge channel.

First, check your clang (just to make sure you will use the new compilers)

clang --version
which clang

Then install compilers from conda

conda install clang_osx-64 clangxx_osx-64 -c anaconda

Now, check the name of the new compiler

ls /Users/jan/anaconda3/bin/ | grep clang

Then update your CC and CXX (use the version seen in the previous step

export CC=x86_64-apple-darwin13.4.0-clang
export CXX=x86_64-apple-darwin13.4.0-clang++
ahartikainen commented 4 years ago

But, if you are already using conda compilers, then I think you are missing a libclang?

conda install libclang -c anaconda
jankaWIS commented 4 years ago

Hm, that is exactly what I was worried of. I had a compiler problem before, gcc and clang. More specifically, I was not able to install gcc from anaconda and I had it from brew but it seemed it does not see each other. Anyway, long message short. I have done this before, I have tried now the installation with conda of the libclang, did not help. The clang I have is the following in zsh:

Apple clang version 12.0.0 (clang-1200.0.32.2)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

and for which:

/usr/bin/clang

However, to not make the things easy, in bash which I usually use it gives me the same version:

Apple clang version 12.0.0 (clang-1200.0.32.2)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

but different location:

/Users/jan/anaconda3/bin/clang

So when I try in bash:

conda install clang_osx-64 clangxx_osx-64 -c anaconda
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

The compilers are:

ls /Users/jan/anaconda3/bin/ | grep clang
clang
clang++
clang-10
clang-cl
clang-cpp
x86_64-apple-darwin13.4.0-clang
x86_64-apple-darwin13.4.0-clang++

And I have updated them in CC and CXX, rebooted but still the same error.

One more note. You have mentioned that an upgrade could cause it. Actually, this laptop is brand new, how could that have happened?

ahartikainen commented 4 years ago

Ok, that is mystery for me then (why it doesn't work).

Can you uninstall all clang/gcc stuff from anaconda and try with the xcode compiler?

jankaWIS commented 4 years ago

So we have tried a few things, at the end what worked was to use clang v10 from Anaconda environment:

export CC=/usr/bin/clang
export CXX=/usr/bin/clang++

Just to make it complete, here it what is there and working:

(base) bash-3.2$ clang --version
clang version 10.0.0
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Users/jan/anaconda3/bin
(base) bash-3.2$ which clang
/Users/jan/anaconda3/bin/clang

So now the code compiles, however, I got stuck on the next step which I believe is connected. If I just use to code to sample:

fit_noLA = stan_model_noLA.sampling(data=data_stan_test, 
                    chains=4, iter=2000, warmup=1000, thin=1, init='random', verbose=True, 
                    control = {"adapt_delta":0.95, "stepsize":1, "max_treedepth":10}, n_jobs=-1)

the code in jupyter runs but never finished and looking at the output in terminal, I see the error ModuleNotFoundError which is then repeating until the ntb saves itself and from terminal looks like it has finished the calculation.

Process SpawnPoolWorker-57:
Process SpawnPoolWorker-58:
Process SpawnPoolWorker-59:
Process SpawnPoolWorker-60:
Traceback (most recent call last):
  File "/Users/jan/anaconda3/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/jan/anaconda3/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/jan/anaconda3/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/jan/anaconda3/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
ModuleNotFoundError: No module named 'stanfit4anon_model_649148968e47447c8e6aa386bb89c275_7077828339563405780'

Any ideas about that? Is it related?

ahartikainen commented 4 years ago

Yes, that is related to python 3.8 + macOS + multiprocessing

Just use this at top of the code

import multiprocessing
multiprocessing.set_start_method("fork")

https://discourse.mc-stan.org/t/pystan-throws-error-when-running-chains-in-parallel-n-jobs-1/17563/4