PCH breaking between calls, rebuild_cmdstan() errors out

[bob-carpenter]

Summary:

Update: It turns out what happened is that in the first report, the first run where it appeared to work, failed in the parser, so it never tried to run the compiler. I had updated Xcode before running everything below, but the problem didn't appear until I finally got around to compiling. So I did pull the rug out from under CmdStan.

This was not at all obvious to me and I was essentially blocked because the usual ways to fix didn't work. I had to reinstall cmdstan (which I did through CmdStanPy). Then everything worked. So I'm just going to close this issue.

Description:

I'm going to give you two session reports. The first one is where the problem crops up. The second is where I fail to fix it.

~/github/bob-carpenter/case-studies/ngrams (master)$ python3 tokenizer.py 
f{np.shape(tokens) = }
Traceback (most recent call last):
  File "/Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/tokenizer.py", line 53, in <module>
    model = csp.CmdStanModel(stan_file = 'bigram.stan')
  File "/usr/local/lib/python3.9/site-packages/cmdstanpy/model.py", line 191, in __init__
    model_info = self.src_info()
  File "/usr/local/lib/python3.9/site-packages/cmdstanpy/model.py", line 305, in src_info
    raise ValueError(
ValueError: Failed to get source info for Stan model '/Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram.stan'. Console:
Semantic error in '/Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram.stan', line 22, column 23 to column 35:
   -------------------------------------------------
    20:    y[1] ~ categorical(theta_uni);
    21:    for (n in 2:N)
    22:      y[n] ~ categorical(theta_bigram[y[n - 1]]);
                                ^
    23:  }
   -------------------------------------------------

Identifier 'theta_bigram' not in scope.

When I try again I get the dreaded PCH error.

~/github/bob-carpenter/case-studies/ngrams (master)$ python3 tokenizer.py 

f{np.shape(tokens) = }

13:46:46 - cmdstanpy - INFO - compiling stan file /Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram.stan to exe file /Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram
13:47:05 - cmdstanpy - WARNING - CmdStan's precompiled header (PCH) files may need to be rebuilt.Please run cmdstanpy.rebuild_cmdstan().
If the issue persists please open a bug report
Traceback (most recent call last):
  File "/Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/tokenizer.py", line 53, in <module>
    model = csp.CmdStanModel(stan_file = 'bigram.stan')
  File "/usr/local/lib/python3.9/site-packages/cmdstanpy/model.py", line 236, in __init__
    self.compile(force=str(compile).lower() == 'force')
  File "/usr/local/lib/python3.9/site-packages/cmdstanpy/model.py", line 563, in compile
    raise ValueError(
ValueError: Failed to compile Stan model '/Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram.stan'. Console:

--- Translating Stan model to C++ code ---
bin/stanc  --o=/Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram.hpp /Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram.stan

--- Compiling, linking C++ code ---
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT -Wno-ignored-attributes      -I stan/lib/stan_math/lib/tbb_2020.3/include    -O3 -I src -I stan/src -I stan/lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.78.0 -I stan/lib/stan_math/lib/sundials_6.1.1/include -I stan/lib/stan_math/lib/sundials_6.1.1/src/sundials    -DBOOST_DISABLE_ASSERTS          -c -include-pch stan/src/stan/model/model_header.hpp.gch -x c++ -o /Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram.o /Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram.hpp
error: PCH file uses an older PCH format that is no longer supported
1 error generated.
make: *** [/Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram] Error 1

Command ['make', '/Users/bcarpenter/github/bob-carpenter/case-studies/ngrams/bigram']
error during processing No such file or directory

~/github/bob-carpenter/case-studies/ngrams (master)$

This is my call to rebuild_cmdstan() failing.

>>> import cmdstanpy as csp

>>> csp.rebuild_cmdstan()
Compiling:  (00:58) |                                                                                                                                                                |  ... Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cmdstanpy/install_cmdstan.py", line 280, in build
    do_command(cmd, fd_out=None, pbar=progress_hook)
  File "/usr/local/lib/python3.9/site-packages/cmdstanpy/utils/command.py", line 76, in do_command
    raise RuntimeError(msg)
RuntimeError: Command ['make', 'build', '-j1']
error during processing No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/cmdstanpy/install_cmdstan.py", line 386, in rebuild_cmdstan
    build(verbose, progress, cores)
  File "/usr/local/lib/python3.9/site-packages/cmdstanpy/install_cmdstan.py", line 286, in build
    raise CmdStanInstallError(f'Command "make build" failed\n{str(e)}')
cmdstanpy.install_cmdstan.CmdStanInstallError: Command "make build" failed
Command ['make', 'build', '-j1']
error during processing No such file or directory
Compiling:  (01:10) |                                                                                                                                                                |  ...
>>> csp.rebuild_cmdstan()
Compiling:  (08:07) |                                                                                                                                                                |  ...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cmdstanpy/install_cmdstan.py", line 280, in build
    do_command(cmd, fd_out=None, pbar=progress_hook)
  File "/usr/local/lib/python3.9/site-packages/cmdstanpy/utils/command.py", line 76, in do_command
    raise RuntimeError(msg)
RuntimeError: Command ['make', 'build', '-j1']
error during processing No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/cmdstanpy/install_cmdstan.py", line 386, in rebuild_cmdstan
    build(verbose, progress, cores)
  File "/usr/local/lib/python3.9/site-packages/cmdstanpy/install_cmdstan.py", line 286, in build
    raise CmdStanInstallError(f'Command "make build" failed\n{str(e)}')
cmdstanpy.install_cmdstan.CmdStanInstallError: Command "make build" failed
Command ['make', 'build', '-j1']
error during processing No such file or directory
Compiling:  (01:06) |                                                                                                                                                                |  ...  
Compiling:  (03:51) |                                                                                                                                                                |  ...
KeyboardInterrupt

>>>

Current Version:

>>> cmdstanpy.show_versions()
INSTALLED VERSIONS
---------------------
python: 3.9.4 (default, Apr  5 2021, 01:47:16) 
[Clang 11.0.0 (clang-1100.0.33.17)]
python-bits: 64
OS: Darwin
OS-release: 22.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
cmdstan_folder: /Users/bcarpenter/.cmdstan/cmdstan-2.31.0
cmdstan: (2, 31)
cmdstanpy: 1.1.0
pandas: 2.0.3
xarray: 2023.1.0
tqdm: 4.64.0
numpy: 1.25.0

' '

[mitzimorris]

we've discussed getting rid of pch headers altogether. time to revisit this issue?

[syclik]

I’m in favor of removal. It’s caused too many headaches to be worth it.

On Thu, Sep 21, 2023 at 2:14 PM Mitzi Morris @.***> wrote:

we've discussed getting rid of pch headers altogether. time to revisit this issue?

— Reply to this email directly, view it on GitHub https://github.com/stan-dev/cmdstanpy/issues/702#issuecomment-1730074055, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADH6F6SLCMY57QKQQ6SEYDX3R7ZBANCNFSM6AAAAAA5B6L4CA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[WardBrian]

Disabling the precompiled headers roughly doubles the amount of time required to compile a model (see https://github.com/roualdes/bridgestan/issues/151). This problem also seems to be essentially exclusive to MacOS, I've never encountered it on Linux

We tried to fix issues like this in https://github.com/stan-dev/cmdstan/pull/1171 which was included in the 2.33 release

[mitzimorris]

This problem also seems to be essentially exclusive to MacOS, I've never encountered it on Linux

I thought it was only implemented for MacOS?

update: defined in CmdStan makefile here: https://github.com/stan-dev/cmdstan/blob/e99e0f2ac9a920624c36899817856183a13d6e3e/makefile#L121-L133

[WardBrian]

We use it on all three major platforms since 2.28

@bob-carpenter can you share the output of csp.rebuild_cmdstan(verbose=True)?

[syclik]

Disabling the precompiled headers roughly doubles the amount of time required to compile a model

Thanks for linking the issue!

I'm always going to favor things that are more robust to configuration error.

Errors like this make the 2x time per compile worth it for me. Whenever I run into it and I have to spend half an hour rebuilding cmdstan to get everything back into a consistent state, it's tough. It's a hard context switch from where I'm normally at. I don't know how someone that doesn't know the infrastructure as well navigates past this problem.

Btw, I'd be happy if there were clear instructions that appear when this error occurs. I just don't think we have that yet.

[WardBrian]

Rebuilding cmdstan usually works (I'm curious what exactly is going wrong for Bob during that).

We used to try to show instructions to do so, but the message the compiler shows when the PCH is bad seems to change often and we ended up writing such a broad net that we were showing that suggestion when the PCH wasn't even the real issue.

Adding PRECOMPILED_HEADERS=false into a make/local file will make it never be used on your individual system, which might be a good idea if you're in a situation where this is happening frequently

[bob-carpenter]

@bob-carpenter can you share the output of csp.rebuild_cmdstan(verbose=True)?

Afraid not. I upgraded CmdStan, updated CmdStanPy, and rebooted. The problem turned out to be because I really had updated in between. I'll update the issue text to reflect that.

I wouldn't want to remove precompiled headers altogether. I'll let you know if I get this problem going forward. If I keep running into the problem, I'll lobby for turning it off by default.

[bob-carpenter]

Closing because we expect PCH to break and require manual fixes when we update the underlying C++ compiler, at least for the version I was using.