walkthough error on Windows 10

[timbp]

Just decided to check out Stan in Julia, but cannot get the walkthough example to work.

julia> rc = stan_sample(sm, data=data);
ERROR: IOError: could not spawn `'C:\Users\MQ2020~1\AppData\Local\Temp\jl_fEP3jt\bernoulli.exe' sample num_samples=1000 num_warmup=1000 save_warmup=0 thin=1 adapt engaged=1 gamma=0.05 delta=0.8 kappa=0.75 t0=10.0 init_buffer=75 term_buffer=50 window=25 algorithm=hmc engine=nuts max_depth=10 metric=diag_e stepsize=1.0 stepsize_jitter=1.0 random seed=-1 init=2 id=1 data 'file=C:\Users\MQ2020~1\AppData\Local\Temp\jl_fEP3jt\bernoulli_data_1.R' output 'file=C:\Users\MQ2020~1\AppData\Local\Temp\jl_fEP3jt\bernoulli_chain_1.csv' refresh=100`: no such file or directory (ENOENT)
Stacktrace:
 [1] _spawn_primitive(::String, ::Cmd, ::Array{Any,1}) at .\process.jl:99
 [2] _spawn(::Cmd, ::Array{Any,1}, ::Base.ProcessChain) at .\process.jl:181
 [3] _spawn(::Base.AndCmds, ::Array{Any,1}, ::Base.ProcessChain) at .\process.jl:174
 [4] #587 at .\process.jl:120 [inlined]
 [5] setup_stdios(::Base.var"#587#588"{Base.AndCmds}, ::Array{Any,1}) at .\process.jl:196
 [6] _spawn(::Base.AndCmds, ::Array{Any,1}) at .\process.jl:119
 [7] _spawn(::Base.CmdRedirect, ::Array{Any,1}) at .\process.jl:139
 [8] run(::Base.CmdRedirect; wait::Bool) at .\process.jl:439
 [9] run at .\process.jl:438 [inlined]
 [10] stan_sample(::SampleModel; kwargs::Base.Iterators.Pairs{Symbol,Dict{String,Any},Tuple{Symbol},NamedTuple{(:data,),Tuple{Dict{String,Any}}}}) at C:\Users\MQ20200577\.julia\packages\StanBase\jHkIX\src\stanrun\stan_sample.jl:126
 [11] top-level scope at REPL[4]:1

I also tried ] test Stan and testing failed with the same error.

[goedman]

Hi Tim, sorry I must have missed the posting of this issue. I don't have access to Windows 10 but others have used StanJulia on Windows. It looks like the .exe file is not present. I'll have a look. I'm assuming you've managed to install and build cmdstan and the Windows tool chains?

[goedman]

Looking back through the issues I believe only CmdStan.jl has been tested on Windows. That should probably be the first one to try yo get working.

[timbp]

Today I uninstalled Julia, R, RTools, and cmdstan, manually removed any leftover directories I could find, and manually removed the relevant path entries, to give as clean a system as possible.

Then.

1. Installed RTools, and confirmed tools are available
2. Built cmdstan (2.26.1) and confirmed I can compile and sample from a model
3. Installed R (4.04) and cmdstanr, and confirmed I can fit a model
4. Installed Julia (1.5.4), CmdStan.jl (6.1.8), and Stan.jl (6.4.0)
5. Tried the walkthrough for Stan.jl -- same error as above
6. Tried the walkthrough for CmdStan.jl -- cancelled the line rc, chms, cnames = stan(stanmodel, bernoullidata, ProjDir, CmdStanDir=CMDSTAN_HOME) after 10 minutes when it didn't seem to be doing anything.

[goedman]

Hi Tim, what do you see after running the first 3 parts in the terminal?

julia> using CmdStan

julia> bernoullistanmodel = "
       data { 
         int<lower=0> N; 
         int<lower=0,upper=1> y[N];
       } 
       parameters {
         real<lower=0,upper=1> theta;
       } 
       model {
         theta ~ beta(1,1);
           y ~ bernoulli(theta);
       }
       "
"\ndata { \n  int<lower=0> N; \n  int<lower=0,upper=1> y[N];\n} \nparameters {\n  real<lower=0,upper=1> theta;\n} \nmodel {\n  theta ~ beta(1,1);\n    y ~ bernoulli(theta);\n}\n"

julia> stanmodel = Stanmodel(name="bernoulli", model=bernoullistanmodel);
File /Users/rob/.julia/dev/CmdStan/tmp/bernoulli.stan will be updated.

[goedman]

In the tmp directory there should be 3 files: bernoulli_build.log, bernoulli_make.log and bernoulli_run.log.

[goedman]

Would you mind to also send me the output of versioninfo()?

[goedman]

You can also just mail to goedman@icloud.com. The build log should be empty and also the run log (as we didn't call stan() yet). The make log should tell me if the toolchain is used correctly.

[timbp]

Just ran it with a completely fresh, clean environment, and the model apparently fit, but then my terminal went weird:

Microsoft Windows [Version 10.0.18362.1256]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\Users\MQ20200577>cd Documents\testcmdstan

C:\Users\MQ20200577\Documents\testcmdstan>ls
Manifest.toml  Project.toml

C:\Users\MQ20200577\Documents\testcmdstan>julia --startup-file=no
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.4 (2021-03-11)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.5) pkg> activate .
 Activating environment at `C:\Users\MQ20200577\Documents\testcmdstan\Project.toml`

(testcmdstan) pkg> status
Status `C:\Users\MQ20200577\Documents\testcmdstan\Project.toml`
  [593b3428] CmdStan v6.1.8

julia> using CmdStan
┌ Warning: Environment variable CMDSTAN_HOME not set. Use set_cmdstan_home!.
└ @ CmdStan C:\Users\MQ20200577\.julia\packages\CmdStan\0wjTa\src\CmdStan.jl:27

julia> set_cmdstan_home!(joinpath("C:\\", "cmdstan", "cmdstan-2.26.1"))
"C:\\cmdstan\\cmdstan-2.26.1"

julia> bernoullistanmodel = "
       data {
         int<lower=0> N;
         int<lower=0,upper=1> y[N];
         }
       parameters {
         real<lower=0,upper=1> theta;
       }
       model {
         theta ~ beta(1,1);
           y ~ bernoulli(theta);
       }
       "
"\ndata { \n  int<lower=0> N; \n  int<lower=0,upper=1> y[N];\n  } \nparameters {\n  real<lower=0,upper=1> theta;\n} \nmodel {\n  theta ~ beta(1,1);\n    y ~ bernoulli(theta);\n}\n"

julia> stanmodel = Stanmodel(name="bernoulli", model=bernoullistanmodel);

File C:\Users\MQ20200577\Documents\testcmdstan\tmp\bernoulli.stan will be updated.

julia> bernoullidata = Dict("N" => 10, "y" => [0, 1, 0, 1, 0, 0, 0, 0, 0, 1])
Dict{String,Any} with 2 entries:
  "N" => 10
  "y" => [0, 1, 0, 1, 0, 0, 0, 0, 0, 1]

julia> ProjDir = @__DIR__
"C:\\Users\\MQ20200577\\Documents\\testcmdstan"

julia> rc, chns, cnames = stan(stanmodel, bernoullidata, ProjDir, CmdStanDir=CMDSTAN_HOME)
Inference for Stan model: bernoulli_model
4 chains: each with iter=(1000,1000,1000,1000); warmup=(0,0,0,0); thin=(1,1,1,1); 4000 iterations saved.

Warmup took (0.0090, 0.0070, 0.0080, 0.013) seconds, 0.037 seconds total
Sampling took (0.021, 0.020, 0.024, 0.019) seconds, 0.084 seconds total

                Mean     MCSE  StdDev    5%   50%   95%    N_Eff  N_Eff/s    R_hat

lp__            -8.2  1.8e-02    0.76  -9.8  -7.9  -7.6     1766    21030     1.00
accept_stat__   0.91  2.0e-03    0.14  0.62  0.97   1.0  4.8e+03  5.7e+04  1.0e+00
stepsize__       1.0  3.9e-02   0.055  0.99   1.0   1.1  2.0e+00  2.4e+01  1.9e+13
treedepth__      1.4  8.1e-03    0.48   1.0   1.0   2.0  3.5e+03  4.2e+04  1.0e+00
n_leapfrog__     2.4  2.0e-02    1.00   1.0   3.0   3.0  2.5e+03  3.0e+04  1.0e+00
divergent__     0.00      nan    0.00  0.00  0.00  0.00      nan      nan      nan
energy__         8.7  2.6e-02     1.1   7.7   8.3    11  1.6e+03  1.9e+04  1.0e+00

theta           0.34  3.9e-03    0.13  0.13  0.33  0.56     1127    13419      1.0

Samples were drawn using hmc with nuts.
For each parameter, N_Eff is a crude measure of effective sample size,
and R_hat is the potential scale reduction factor on split chains (at
convergence, R_hat=1).
←[0m(0, [-7.67708 1.0 … 7.71613 0.29615; -8.21778 0.870453 … 8.5206 0.486997; … ; -7.69171 0.744134 … 8.40861 0.37876; -7.93575 0.918274 … 8.11069 0.234994]

[-8.04717 0.889514 … 8.16922 0.219626; -7.6467 0.928438 … 8.60194 0.351258; … ; -7.639 1.0 … 8.60263 0.338908; -7.9985 0.780232 … 8.80762 0.225979]

[-8.37497 0.541489 … 11.4051 0.507003; -7.98489 1.0 … 8.305 0.451434; … ; -7.67345 0.991695 … 7.74065 0.29789; -7.64362 0.999993 … 7.66614 0.319225]

[-8.3376 0.964985 … 8.376 0.502462; -8.04985 1.0 … 8.32469 0.462308; … ; -7.76705 0.939421 … 8.08445 0.266964; -7.63914 1.0 … 7.75709 0.327355], ["lp__", "accept_stat__", "stepsize__", "treedepth__", "n_leapfrog__", "divergent__", "energy__", "theta"])

←[7C←[32m←[1mjulia> ←[0m←[0m

[timbp]

julia> versioninfo()
Julia Version 1.5.4
Commit 69fcb5745b (2021-03-11 19:13 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)

julia>

[timbp]

bernoulli.exe_make.log

[goedman]

Yes the make (and the run) logs will be ok because the model works fine. At the end the call to stan() calls a Stan executable called stansummary which creates the summary at the end of your first post. Unfortunately that summary has make-up characters in it (at the start of the chains ←[0m and several more at the end of printing the summary. I'll take a look at that. In Stan.jl I suppress that by default.

Can you run the call to stan() with summary=false:

rc, chns, cnames = stan(stanmodel, bernoullidata, ProjDir, CmdStanDir=CMDSTAN_HOME, summary=false)

If that' s ok with you, I'll also send you a small script tomorrow to run using Stan.jl packages and figure out why that doesn't work. In general it's not a good idea to mix and match e.g. CmdStan.jl and StanSample.jl.

I'm also a bit surprised versioninfo() doesn't show a JULIA_CMDSTAN_HOME, e.g.

Julia Version 1.7.0-DEV.705
Commit a43e01f787* (2021-03-12 23:26 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin20.4.0)
  CPU: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)
Environment:
  JULIA_EDITOR = subl
  JULIA_CMDSTAN_HOME = /Users/rob/Projects/StanSupport/cmdstan
  JULIA_SPECIALFUNCTIONS_BUILD_SOURCE = true
  JULIA_PKG3_PRECOMPILE = true
  JULIA_PKG_SERVER = https://us-east.pkg.julialang.org

julia> 

In that case you don't need set_cmdstan-home!().

Do you use the PowerShell?

[timbp]

OK. It appears to be the shell. I don't usually use Powershell.

I usually use the git bash shell. Today, to keep things as standard as possible, I used the windows Command Prompt.

I now confirmed the walkthroughs for both CmdStan and StanSamples work if I start julia from Command Prompt or from Powershell, but fail if I start julia from git bash.

[goedman]

This is good news! Thanks for sticking with it.

I definitely need to update the docs, almost 10 years is a long time, about the time I wrote the first version. But mainly need to take activate & co into consideration. I might postpone that though until StatisticalRethinking is completed as I'm sure working with multilevel models will lead to further adjustment in StanJulia (as the default for read_samples() in StanJulia is now a NamedTuple that organizes parameters such as "a.1", "a.2", etc better.

[goedman]

Hi Tim ( @timbp ), one more question, do you use VSCode or another development environments (e.g. Julia REPL + an editor)? I'm trying to help out here.

[timbp]

I use both VSCode and REPL+Editor, depending on what I'm doing. All my testing last week was REPL+Editor.

I can confirm CmdStan messes up the Julia terminal in both VSCode and REPL in Command Prompt or Powershell. The StanSample approach doesn't cause problems.

Also, I tried to dev Stan but got a error about invalid path.

ERROR: failed to clone from https://github.com/StanJulia/Stan.jl.git, 
error: GitError(Code:ERROR, Class:Checkout, cannot checkout to invalid 
path 'Examples_DiffEqBayes/Predator-Prey Population Dynamics: the 
Lotka-Volterra model in Stan.pdf')

[goedman]

Thanks a lot Tim. Glad to hear StanSample + VSCode + Julia REPL works ok. I'll continue to look what the problem with CmdStan.jl is but slowly the picture gets a bit clearer.

I've just release a new version of Stan.jl. The problem was the blanks in the filename pointing to a pdf file. I've removed that link and that should fix your issue. The versions are:

(Stan) pkg> st
     Project Stan v6.4.6
      Status `~/.julia/dev/Stan/Project.toml`
  [336ed68f] CSV v0.8.4
  [a93c6f00] DataFrames v0.22.5
  [31c24e10] Distributions v0.24.15
  [ffbed154] DocStringExtensions v0.8.3
  [d0ee94f6] StanBase v2.2.6
  [fb13fc95] StanDiagnose v2.2.4
  [fbd8da12] StanOptimize v2.2.4
  [e4723793] StanQuap v1.0.2
  [c1514b29] StanSample v3.0.10
  [6ede68ce] StanVariational v2.2.4
  [9a3f8284] Random
  [8dfed614] Test

Hope that works for you, thanks again for your help, Rob