Appending ".exe" on Windows

[jwright11]

I'm trying to get cmdstan.jl working on Windows10 and Julia 1.5.3. I've installed cmdstan and can successfully use it from the command line. I'm running into an issue with the bernoulli.jl file and get the following error:

julia> sm = SampleModel("bernoulli", bernoullimodel,
         method=StanSample.Sample(save_warmup=true, num_warmup=1000,
         num_samples=1000, thin=1, adapt=StanSample.Adapt(delta=0.85)),
         tmpdir=tmpdir);

C:\Users\wrighjo\AppData\Local\Temp\jl_jJ25Jo\bernoulli.stan updated.
ERROR: 
Error when compiling SampleModel bernoulli:
make: *** No rule to make target 'C:\Users\wrighjo\AppData\Local\Temp\jl_jJ25Jo\bernoulli'.  Stop.

Per the cmdstan installation instructions, I have to append ".exe" to the end of the command to compile the bernoulli stan file. I went to the command line to see if I can compile it there and it works:

C:\cmdstan>make C:\Users\wrighjo\AppData\Local\Temp\jl_jJ25Jo\bernoulli.exe

--- Translating Stan model to C++ code ---
bin/stanc.exe  --o=C:/Users/wrighjo/AppData/Local/Temp/jl_jJ25Jo/bernoulli.hpp C:/Users/wrighjo/AppData/Local/Temp/jl_jJ25Jo/bernoulli.stan

--- Compiling, linking C++ code ---
g++ -std=c++1y -m64 -D_REENTRANT -Wall -Wno-unused-function -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-variable -Wno-sign-compare -Wno-unused-local-typedefs -Wno-int-in-bool-context -Wno-attributes -Wno-ignored-attributes      -I stan/lib/stan_math/lib/tbb_2019_U8/include   -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.72.0 -I stan/lib/stan_math/lib/sundials_5.6.1/include  -D_USE_MATH_DEFINES  -DBOOST_DISABLE_ASSERTS         -c  -x c++ -o C:/Users/wrighjo/AppData/Local/Temp/jl_jJ25Jo/bernoulli.o C:/Users/wrighjo/AppData/Local/Temp/jl_jJ25Jo/bernoulli.hpp
g++ -std=c++1y -m64 -D_REENTRANT -Wall -Wno-unused-function -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-variable -Wno-sign-compare -Wno-unused-local-typedefs -Wno-int-in-bool-context -Wno-attributes -Wno-ignored-attributes      -I stan/lib/stan_math/lib/tbb_2019_U8/include   -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.72.0 -I stan/lib/stan_math/lib/sundials_5.6.1/include  -D_USE_MATH_DEFINES  -DBOOST_DISABLE_ASSERTS               -Wl,-L,"/c/cmdstan/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/c/cmdstan/stan/lib/stan_math/lib/tbb"      C:/Users/wrighjo/AppData/Local/Temp/jl_jJ25Jo/bernoulli.o src/cmdstan/main.o  -static-libgcc -static-libstdc++       stan/lib/stan_math/lib/sundials_5.6.1/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_5.6.1/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_5.6.1/lib/libsundials_idas.a stan/lib/stan_math/lib/sundials_5.6.1/lib/libsundials_kinsol.a  stan/lib/stan_math/lib/tbb/tbb.dll -o C:/Users/wrighjo/AppData/Local/Temp/jl_jJ25Jo/bernoulli.exe
rm -f C:/Users/wrighjo/AppData/Local/Temp/jl_jJ25Jo/bernoulli.o

Could this be a case of cmdstan.jl not appending ".exe" to the end of the command on Windows machines?

[goedman]

Absolutely. I'll take a closer look.

[goedman]

Starting on lines 108 in stancode.jl:

    if @static Sys.iswindows() ? true : false
      tmpmodelname = replace(tmpmodelname*".exe", "\\" => "/")
    end

.exe should have been added but apparently isn't. Unfortunately I can't check that on my system.

Would you be able to install Cmdstan.jl in ~/.julia/dev, e.g.:

julia> ]
(CmdStan) pkg> dev CmdStan

If that works, a simple println(tmpmodelname) before the line with end should tell us if that is where the error comes from, e.g. maybe in @static Sys,windows() has changed?

[goedman]

Actually, you are running bernoulli.jl from examples? Doesn't matter really, can you send me the contents of C:/Users/wrighjo/AppData/Local/Temp/jl_jJ25Jo/bernoulli_make.log?

[jwright11]

I don't have a bernoulli_make.log file in that folder. I have three files: bernoulli.stan, bernoulli.exe, and bernoulli.hpp. The exe and hpp files are from me running cmdstan from the command line. Yes, I used bernoulli.jl from examples. I'll try the dev thing in a little bit.

[jwright11]

Sorry but I don't know how to install cmdstan.jl into that dev folder (still pretty new to julia). Can you give some more instructions please?

I did just try running

julia> Sys.iswindows()
true

And just to clarify, this is the script I'm trying to run:

######### Stan program example  ###########

using StanSample, MCMCChains
using StatsPlots

bernoullimodel = "
data { 
  int<lower=1> N; 
  int<lower=0,upper=1> y[N];
} 
parameters {
  real<lower=0,upper=1> theta;
} 
model {
  theta ~ beta(1,1);
  y ~ bernoulli(theta);
}
";

observed_data = Dict("N" => 10, "y" => [0, 1, 0, 1, 0, 0, 0, 0, 0, 1])

# Default for tmpdir is to create a new tmpdir location
# To prevent recompilation of a Stan progam, choose a fixed location,
tmpdir=mktempdir()

sm = SampleModel("bernoulli", bernoullimodel,
  method=StanSample.Sample(save_warmup=true, num_warmup=1000, 
  num_samples=1000, thin=1, adapt=StanSample.Adapt(delta=0.85)),
  tmpdir=tmpdir);

rc = stan_sample(sm, data=observed_data);

if success(rc)
  chns = read_samples(sm; output_format=:mcmcchains)

  # Describe the results
  chns

  # Optionally, read samples as a a DataFrame
  df=read_samples(sm, output_format=:dataframe)
  first(df, 5)
  println()

  # Look at effective sample saize
  ess(chns) |> display
  println()

  # Check if StatsPlots is available and show basic MCMCChains plots
  if isdefined(Main, :StatsPlots)
    cd(@__DIR__) do
      p1 = plot(chns)
      savefig(p1, joinpath(tmpdir, "traceplot.pdf"))
      p2 = pooleddensity(chns)
      savefig(p2, joinpath(tmpdir, "pooleddensity.pdf"))
    end
  end

  df = read_summary(sm)
  df[df.parameters .== :theta, [:mean, :ess]]
end

[goedman]

let's go step by ste.

1. You can try it in the REPL:

   
   julia> tmpmodelname = homedir()*"/bernoulli"

julia> if @static Sys.iswindows() ? true : false tmpmodelname = replace(tmpmodelname*".exe", "\" => "/") println(tmpmodelname) end

This should return something like `/Users/rob/bernoulli.exe`.

2. To install CmdStan.jl in development mode you can also type `Pkg.develop("CmdStan")` in the REPL.

3. I'm confused why there is no `bernoulli_make.log` file in the temp dir.

If you include the example, `bernoulli.jl` into the REPL, it will print out which temp directory it is using, in your case something like:

C:\Users\wrighjo\AppData\Local\Temp\jl_jJ25Jo\bernoulli.stan updated.

In that directory should be a bernoulli_make.log file.

Let me know if this is a bit clearer.

[jwright11]

Ah, I think maybe we need: "/" => "\\"

julia> tmpmodelname = homedir()*"/bernoulli"
"C:\\Users\\wrighjo/bernoulli"

julia> if @static Sys.iswindows() ? true : false
                tmpmodelname = replace(tmpmodelname*".exe", "\\" => "/")
                println(tmpmodelname)
                       end
C:/Users/wrighjo/bernoulli.exe

julia> tmpmodelname
"C:/Users/wrighjo/bernoulli.exe"

But if we do this:

julia> tmpmodelname
"C:/Users/wrighjo/bernoulli.exe"

julia> tmpmodelname = homedir()*"/bernoulli"
"C:\\Users\\wrighjo/bernoulli"

julia> if @static Sys.iswindows() ? true : false
                tmpmodelname = replace(tmpmodelname*".exe", "/" => "\\")
                println(tmpmodelname)
                       end
C:\Users\wrighjo\bernoulli.exe

julia> tmpmodelname
"C:\\Users\\wrighjo\\bernoulli.exe"

[goedman]

Hmm, in that case, will the double \\ work with your above make command:

C:\cmdstan>make C:\\Users\\wrighjo\\AppData\Local\\Temp\jl_jJ25Jo\\bernoulli.exe

[goedman]

Were you able to install the development version?

[jwright11]

Dang, maybe that's not it. It seems like all of these variants are working:

C:\cmdstan>make C:\\Users\\wrighjo\\AppData\\Local\\Temp\\jl_jJ25Jo\\bernoulli.exe
make: 'C:\\Users\\wrighjo\\AppData\\Local\\Temp\\jl_jJ25Jo\\bernoulli.exe' is up to date.

C:\cmdstan>make C:\Users\wrighjo\AppData\Local\Temp\jl_jJ25Jo\bernoulli.exe
make: 'C:\Users\wrighjo\AppData\Local\Temp\jl_jJ25Jo\bernoulli.exe' is up to date.

C:\cmdstan>make C:/Users/wrighjo/AppData/Local/Temp/jl_jJ25Jo/bernoulli.exe
make: 'C:/Users/wrighjo/AppData/Local/Temp/jl_jJ25Jo/bernoulli.exe' is up to date.

Yes, I just got the development version installed by doing using Pkg Pkg.develop("CmdStan")

[jwright11]

Perhaps I posted this in the wrong place. If I go to the SampleModel.jl file in StanSample.jl, starting on line 41:

function SampleModel(
  name::AbstractString,
  model::AbstractString,
  n_chains=[4];
  seed = StanBase.RandomSeed(),
  init = StanBase.Init(),
  output = StanBase.Output(),
  tmpdir = mktempdir(),
  method = Sample(),
)

  !isdir(tmpdir) && mkdir(tmpdir)

  StanBase.update_model_file(joinpath(tmpdir, "$(name).stan"), strip(model))

  output_base = joinpath(tmpdir, name)
  exec_path = output_base
  cmdstan_home = get_cmdstan_home()

  error_output = IOBuffer()
  is_ok = cd(cmdstan_home) do
      success(pipeline(`make -f $(cmdstan_home)/makefile -C $(cmdstan_home) $(exec_path)`;
                       stderr = error_output))
  end

Line 56 is output_base = joinpath(tmpdir, name) which does not appear to be trying to append ".exe". Maybe we need to add similar logic with the Sys.iswindows() stuff?

[goedman]

Cool, I expect the package CmdStan can be found with something like:

pathof(CmdStan)

If that is correct, the lines I mentioned earlier are in "that directory"/src/stancode.jl.

If you make a change, you need to restart Julia to compile the new version.

The other line of attack is to find the bernoulli_make.log file we can see if .exe is appended.

Yes, I never had someone to test out StanSample.jl on Windows. If you prefer to use StanSample.jl I'll check that tomorrow.

[goedman]

Oops, I should have noticed that from your initial post.

I suggest we close this issue and open an issue on StanSample.jl.

Given that you have CmdStan.jl now installed, if you want we can quickly check if that works.

Assuming you are in Julia REPL:

julia> Pkg.test("CmdStan")
# Lots of output.
...

I had a quick look at StanSample and adding .exe is done in StanBase.jl. I recently had to make a few updates, so maybe I introduced a problem. If you are interested to proceed, let's continue open the StanSample.jl issue mentioned above.

[jwright11]

Sorry, pretty new to Stan and Julia. I pulled the bernoulli example from cmdstan.jl so thought this was the place to post. I will close here and open on stansamples.jl later today/tonight. Definitely interested in pursuing further and really appreciate your help!