Package does not compile in Julia 1.3 (Windows)

[sdewaele]

I am trying to use CmdStan 5.5.0 in Julia 1.3 in Windows. However, when I try to use the package, compilation does not seem to complete.

julia>versioninfo()
Julia Version 1.3.1
Commit 2d5741174c (2019-12-30 21:36 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Environment:
  JULIA_CMDSTAN_HOME = C:\Users\userid\cmdstan

julia> using CmdStan

The last command does not return for at least 20 minutes. When I try to compile CmdStan on the same system with Julia 1.1, compilation is successful. In an attempt to fix the issue, I removed some dependencies (see #78), but this did not resolve this issue. Hence, I am filing this issue in addition to #78.

[itsdfish]

Rob, CmdStan .5.5.0 works on Ubuntu with 1.3 and 1.3.1. So this might indeed be a windows problem.

[goedman]

Thanks Stijn and Chris,

Sorry for this long, and not very to the point, answer.

Yes, I have noticed significant problems as well. I suspect this has partially to do with the new threads model, but not only that. Ever since the very first version of Stan.jl I defined * as concatenation for commands. Somehow this no longer works reliably on 1.3 and up (or takes a really long time to compile). Not receiving any output on Travis has become a regular error, even if all tests succeed, docs might fail with that error.

On OSX the situation is slightly better, but sometimes it takes minutes to complete I think the 2nd part of compilation. Occasionally it simply hangs (and a Julia thread is left hanging, certainly taking many CPU cycles, but seemingly never returning to the REPL). In many ways it reminds me when we switched Java to green threads in the early 90s, hence my earlier sentence.

In v5.5 I took the * based command concatenation out and replaced it by command interpolation, but never really trusted that solved the problem in Julia 1.3 (I think I mentioned this once to you Chris). But it got me through Travis and helped out the Turing folks.

Currently I'm studying similar issues in the updated Stan.jl, where I use Tamas' much cleaner pmap() approach. But also in that setup, I'm running into weird issues.

Ever since the versions of 1.3 which included the new thread model, I've spent many hours trying to resolve the issues without much luck. And unfortunately, the problem is not easily captured in a MWE. Just for completeness, I'm also running into decoupled thread issues with say Plots/GR.

So both this issue and Stijn's other issue (removing superfluous dependencies) are spot on but I don't have a concrete solution yet. I can remove StatsPlots, but won't it be re-introduced through MCMCChains.jl?

[sdewaele]

Thanks for pointing at the underlying issues. It is too bad that there is no easy fix.

I am just curious whether you ever looked at using the in-memory interface that PyStan and RStan use? I think these do not require as much writing of files and calling external executables, so maybe it is more suitable for a multi-threading environment. I realize that this is a big effort, but since it seems that you plan to continue development on Stan.jl/CmdStan.jl, it may be worth a consideration.

BTW - Thanks for your work on the Julia - Stan interface, I appreciate this very much!

[goedman]

Stan.jl got started as a replacement for Jags to play around with the Gelman/.Hill and Bolstad books (now shifted to StatisticalRethinking). Coming from R it felt natural to do the heavy lifting in C/C++ or Fortran. At some point there were some issues with compiling cmdstan using gcc vs. clang and while trying to help with that, simultaneously I took a closer look at the C++ route. But decided against it, partially because I wanted to stay a user of Julia (I am not a programmer) and also because Cxx.jl in those days was not yet as reliable as right now.

While working on StatisticalRethinking I am exploring a few more options. My experience has been that MCMCChains is great for mcmc diagnostics but as soon as I want to work with the data, I convert to a DataFrame where parameters simply become columns, not hidden in an AxisArray. In some cases I have started to use Particles.jl.

For Stan.jl this could very well lead to a decision to drop MCMCChains from the core packages (StanBase, StanSample, etc.) and offer it in a separate package (StanDiagnostics?). This will address #78 at least to some extent. It won't help for #79. None of this affects/helps CmdStan.jl of course.

[goedman]

Hi Stijn, over the last 2 weeks I worked on stuff maybe related to this issue. I'm assuming CmdStan.jl is still not compiling on Julia 1.3 on Windows?

If not, I wonder if you would be willing to test StanBase.jl:

Pkg.add("StanBase")
] test StanBase

and send me the output.

[sdewaele]

I cannot replicate the issue even though I am still using CmdStan v5.5.0.

julia> versioninfo()
Julia Version 1.3.1
Commit 2d5741174c (2019-12-30 21:36 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Environment:
  JULIA_CMDSTAN_HOME = C:\Users\marionstijn\cmdstan
  JULIA_EDITOR = "C:\Users\userid\AppData\Local\Programs\Microsoft VS Code\Code.exe"
  JULIA_NUM_THREADS = 2

julia> @time using CmdStan
[ Info: Precompiling CmdStan [593b3428-ca2f-500c-ae53-031589ec8ddd]
118.796075 seconds (29.77 M allocations: 1.472 GiB, 0.50% gc time)

julia> @time using CmdStan
 15.362910 seconds (29.57 M allocations: 1.464 GiB, 3.92% gc time)

I get similar results with 1 thread. The school8.jl example also runs now:

julia> @time include("C:\\Users\\userid\\.julia\\packages\\CmdStan\\Kh1M6\\examples\\EightSchools\\schools8.jl")
[ Info: Precompiling StatsPlots [f3b207a7-027a-5e70-b257-86293d7955fd]

File C:\Users\userid\AppData\Local\Temp\jl_Tuilxg\schools8.stan will be updated.
...
181.792435 seconds (203.40 M allocations: 10.954 GiB, 1.99% gc time)

Second run, without StatsBase precompilation / stan compilation:

25.092965 seconds (4.47 M allocations: 492.133 MiB, 0.37% gc time)

[sdewaele]

When I run test StanBase for v1.1.6, I get the error:

make: *** No rule to make target 'C:\Users\userid\AppData\Local\Temp\jl_6YYneZ\help'.  Stop.

Stacktrace:
 [1] #HelpModel#6(::StanBase.RandomSeed, ::StanBase.Init, ::StanBase.Output, ::String, ::StanBase.Help, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Type{HelpModel}, ::String, ::String, ::Array{Int64,1}) at C:\Users\userid\.julia\packages\StanBase\7p04S\src\stanmodel\HelpModel.jl:34
 [2] HelpModel at C:\Users\userid\.julia\packages\StanBase\7p04S\src\stanmodel\HelpModel.jl:20 [inlined] (repeats 2 times)
 [3] top-level scope at C:\Users\userid\.julia\packages\StanBase\7p04S\test\runtests.jl:24
 [4] include at .\boot.jl:328 [inlined]
 [5] include_relative(::Module, ::String) at .\loading.jl:1105
 [6] include(::Module, ::String) at .\Base.jl:31
 [7] include(::String) at .\client.jl:424
 [8] top-level scope at none:6

Note that in the same environment, CmdStan does run successfully. BTW - I have started using Stan through PyCall/PyStan; this works quite well!

[sdewaele]

Closing since I can't reproduce the issue.

[Byrth]

I can reproduce a form of this issue on Ubuntu 18.04 with Julia 1.3.0 or 1.3.1. CmdStan does still load, but it causes massive input lag (10 minutes) for the next command after load (and thus is probably still processing the prior command.)

This is caused by the NamedArrays dependency. I cannot definitively state what is wrong with it, but I opened an issue on the repository: https://github.com/davidavdav/NamedArrays.jl/issues/81

In addition, I spent some time reviewing the CmdStan.jl code and think these lines maybe could be eliminated:

• @reexport using MCMCChains, Statistics on line 4 of CmdStan.jl
• import Base:* on line 1 of stancode.jl exists but, as noted in the posts above, is unused.

[goedman]

Hi @Byrth thanks for contributing and pinpointing a possible way out.

I'll see if I can reproduce your results after removing NamedArrays and above cleanups.

[Byrth]

No problem! As far as I could tell, NamedArrays were only used in one signature of convert_a3d(). Unfortunately, there is no real way to drop them while retaining the functionality because the functionality is defined by converting to the NamedArray type.

[goedman]

Agreed, it will have to be a major upgrade to v6.0.0. I'm certainly willing to drop NamedArrays. If needed it could easily be generated from the .csv files in a separate package like StanMamba.

To a certain extent I'm also running into these kind of long delays in Stan.jl and have noticed removing MCMCChains makes a huge difference. I don't think anything is wrong with MCMCChains, but is has many dependencies.

When testing, in all these cases Travis seems always trying to resolve package versions.

[Byrth]

If you want a comparable data type, a NamedTuple of Arrays would be pretty analogous to a NamedArray, as I understand it, and NamedTuple is part of Base now.

MCMCChains is another convert_a3d()-only dependency (although it could remain a test/example dependency.) Removing optional conversion to other formats would eliminate both problems, but I am biased because I just use the :array option all the time.

I don't know what is paradigmatic in Julia. This seems like one of those edge cases where it doesn't make sense for the home packages to have conversion from the 3D array format to their format in the particular way required, but it also doesn't make sense to require and extend their methods unless the user is already using them. I will ask around and see if there is a preferred way to handle cases like this.

[Byrth]

https://github.com/MikeInnes/Requires.jl Looks like this is it at the moment. It has a few issues, detailed in the below link, the most serious of which is probably the lack of [compat] compatibility in Project.toml. The below link indicates that there may be a Base solution may be somewhere in development.

https://github.com/JuliaLang/Pkg.jl/issues/1285

By the way, after commenting out the convert_a3d() definitions, this top of CmdStan.jl works for me:

module CmdStan

#using NamedArrays # convert_a3d.jl
using DelimitedFiles # create_r_files.jl
using Unicode # read_optimize.jl and read_variational.jl
#using MCMCChains # convert_a3d.jl
using DataFrames # read_summary.jl
using CSV # read_summary.jl, but not some of the other places it could probably be used (run_variational.jl, stan_summary.jl)

#@reexport using MCMCChains, Statistics

[goedman]

Currently on CmdStan#master is a version very close to yours and it seems to build fine on Travis. I find it hard to quantify how much faster it is, but it certainly seems faster with test CmdStan on my system. Also removed the import Base.* and dependencies on NamedArrays.

I might have made a mistake when I switched by default over to MCMCChains in v5.0.0. Maybe the correct way out is dropping MCMCChains from CmdStan (thus reverting back to array output by default) and fall back to StanMCMCChains.jl to reintroduce MCMCChains if so desired. That is in fact how it was in v4.5.0. If NamedArrays or NamedTuples is used/desired, I can add those in a similar way.

I'll think about it a bit more, but that might be a good choice for v6.0.0.

[Byrth]

Yeah, that sounds like a good choice to me, but I defer to #master

[sdewaele]

@Byrth thanks for looking into this more closely. I have learned a lot from the links you shared. I think properly handling glueing packages (e.g. defining conversion to MCMCChains for CmdStan output) can bring Julia to the next level.

You mentioned that you are quite new to Julia. I hope that you are still seeing benefits in Julia despite issues like this one. Mostly, Julia packages work fine, and code actually run fast! Plus, if you are so new to it, you are a fast learner to be able to pinpoint issues like this.

[Byrth]

@sdewaele Oh, I'm still happy to ride on the rising edge of this Julia wave and have found the community to be great and surprisingly responsive. I am doing my current project in Julia, so I am bought in for at least one more doubling of userbase.

I enjoy the basic concept of Julia (if I can oversimplify, trying to imagine a high-level language that matches machine logic instead of trying to write machine logic to match human language) and think it makes sense.

I do think there are some implementation hurdles (all that compiles is not fast), but I think of the language as more of a competitor to/replacement for C# at the moment than R or Python. The average optimal Julia code is likely slower than optimal C, but much faster than a pure scripting language. I enjoy the workflow where you can start developing things in julia and then, when you want to productionalize it, you just start profiling your julia code and fixing problems instead of being forced to rewrite everything in C (Cython, Rcpp).

[goedman]

Thank you both ( @sdewaele and @Byrth ) for your suggestions/observations. I need to update the documentation but have used Requires.jl in the upcoming v6.0.0 release of CmdStan.jl. (CmdStan.jl#master contains that version). Ultimately I think the majority of the compilation delay was caused by the dependencies of MCMCChains. I'm shooting for a merge/release by the weekend.

[Byrth]

No problem and looks good! Thanks for the rapid release.

If I was going to make any more suggestions about Project.toml, they would be: 1) Test could be under an [extras] heading 2) Documenter could be removed from the main Project.toml because the one in docs/ defines the environment that will be used when making it, right?