Big precompilation regression on v0.6.1

[Paalon]

I am new to the Julia and I use DifferentialEquations but it takes too long time (more than 10 minutes) to import. Did I make a mistake? Is there any way to shorten or deal with this? Thank you.

[ChrisRackauckas]

The first time it will compile all of the packages. It should say in the REPL that it's precompiling the whole thing, and that will take a bit. However, all subsequent using DifferentialEquations will not have to do that step (until the package is updated).

[Paalon]

Does the first time mean the first time of every Julia session?

[ChrisRackauckas]

no, the first time the package is used after it's installed or updated.

[Paalon]

I encountered taking long time for every Julia sessions.

[ChrisRackauckas]

Really? It should only precompile the first time and then be instant after that first time. Are you using a standard installation?

[Paalon]

My environment is: macOS Sierra 10.12.6 I installed Julia through Homebrew:

$ brew cask install julia

I add the DifferentialEquations package in the REPL Julia

$ julia

Pkg.add("DifferentialEquations")
using DifferentialEquations

It takes long time for every sessions at using DifferentialEquations.

[ChrisRackauckas]

Can I get your versioninfo() output?

[Paalon]

This.

julia> versioninfo()
Julia Version 0.6.1
Commit 0d7248e (2017-10-24 22:15 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, ivybridge)

[ChrisRackauckas]

Thanks. I haven't upgraded to v0.6.1. I'll check if that introduced some kind of compilation issue.

[ChrisRackauckas]

Yes, I can reproduce this. This is really bad.

On v0.6.0, precompilation was long:

julia> @time using DifferentialEquations
INFO: Precompiling module DifferentialEquations.
WARNING: using LinAlg.Hessenberg in module IterativeSolvers conflicts with an existing identifier.
464.069996 seconds (10.97 M allocations: 776.638 MiB, 0.09% gc time)

but then subsequent usings were managable:

julia> @time using DifferentialEquations
  6.936111 seconds (11.42 M allocations: 802.791 MiB, 7.63% gc time)

However, on v0.6.1 there's a regression in both. Precompilation time is... not good...:

julia> @time using DifferentialEquations
INFO: Precompiling module DifferentialEquations.
WARNING: using LinAlg.Hessenberg in module IterativeSolvers conflicts with an existing identifier.
1727.608510 seconds (17.48 M allocations: 1.284 GiB, 0.05% gc time)

but it doesn't seem to be precompiling anything since subsequent usings are now what precompilation used to be:

julia> @time using DifferentialEquations
442.447126 seconds (17.43 M allocations: 1.282 GiB, 0.18% gc time)

I'm doing this on Windows, so that means Mac+Windows this is pretty universal. I'll get in contact with the compiler people to try and pin down what's going on, but for now I am sorry to say that I don't have a good solution other than to avoid v0.6.1. Thank you for informing us about this issue.

[ChrisRackauckas]

@Paalon the workaround for now is to use the low dependency version of DiffEq:

http://docs.juliadiffeq.org/stable/features/low_dep.html

All of DifferentialEquations.jl is implemented in separate packages (except for the automatic algorithm selection), and so you can still use the ODE solvers via using OrdinaryDiffEq, or the SDE solvers by using StochasticDiffEq, etc. and those will be much much quicker to load.

[Paalon]

Thank you for your immediate response. I'm looking forward to the release of Julia v1.0. using OrdinaryDiffEq makes me temporarily happy. Thank you!

[mauro3]

I don't see this on Linux. For the precompilation it took about 4 minutes (I didn't @time it). Subsequently:

   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.1 (2017-10-24 22:15 UTC)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |  x86_64-pc-linux-gnu

julia> @time using DifferentialEquations                                                                                                                                 
  4.009507 seconds (6.20 M allocations: 496.479 MiB, 4.38% gc time)                                                                                                      

[ChrisRackauckas]

Thanks, so that means it's OS specific? @mauro3 did you use the generic binaries?

[mauro3]

did you use the generic binaries?

No, I compiled Julia.

[ChrisRackauckas]

I am using generic binaries. I think Homebrew installs generic binaries. It may be something to do with that.

[mauro3]

I think this may be DifferentialEquations v3.0.0 specific: the timings I reported were for v2.3.0. Currently precompiling v3 and it is taking loooooong. I guess I'll report in 20min how long it took...

[ChrisRackauckas]

I think I have narrowed it down to BoundaryValueDiffEq.jl. It seems like it is having issues all on its own. As for why... it's a tiny little library. But what's odd is that its precompile statement is commented out. It'll take awhile to know for sure (lol), but that one commented precompilation statement could cause this whole thing to hit an obscure bug.

[ChrisRackauckas]

Yup,

julia> @time using DifferentialEquations
INFO: Precompiling module DifferentialEquations.
WARNING: using LinAlg.Hessenberg in module IterativeSolvers conflicts with an existing identifier.
213.212237 seconds (6.38 M allocations: 506.605 MiB, 0.17% gc time)

and then new REPL:

julia> @time using DifferentialEquations
  4.261012 seconds (6.33 M allocations: 503.438 MiB, 7.02% gc time)

that's v0.6.1 without BoundaryValueDiffEq.jl. So it's one library... we're close...

[mauro3]

And BoundaryValueDiffEq was fast to compile on Julia 0.6.0? It was not part of DifferentialEquations v2.3.0.

[Paalon]

On Linux with binary distributed Julia v0.6.1:

julia> versioninfo()
Julia Version 0.6.1
Commit 0d7248e2ff (2017-10-24 22:15 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i5-5250U CPU @ 1.60GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, broadwell)

First using with precompiling:

julia> @time using DifferentialEquations
INFO: Precompiling module DifferentialEquations.
WARNING: using LinAlg.Hessenberg in module IterativeSolvers conflicts with an existing identifier.
1952.414733 seconds (15.75 M allocations: 1.190 GiB, 0.04% gc time)

Second using:

julia> @time using DifferentialEquations
467.480945 seconds (15.71 M allocations: 1.188 GiB, 0.19% gc time)

[ChrisRackauckas]

Gottem:

julia> @time using BoundaryValueDiffEq
INFO: Precompiling module BoundaryValueDiffEq.
2128.440347 seconds (1.65 G allocations: 78.449 GiB, 0.45% gc time)

and the second time

julia> @time using BoundaryValueDiffEq
1296.676358 seconds (1.65 G allocations: 78.445 GiB, 0.70% gc time)

The hotfix is to remove this one library.

[ChrisRackauckas]

julia> @time using DifferentialEquations
INFO: Precompiling module DifferentialEquations.
WARNING: using LinAlg.Hessenberg in module IterativeSolvers conflicts with an existing identifier.
171.567514 seconds (6.36 M allocations: 502.867 MiB, 0.17% gc time)

julia> @time using DifferentialEquations
  4.443462 seconds (6.30 M allocations: 499.921 MiB, 6.67% gc time)

on master without BoundaryValueDiffEq.jl. I'll hotfix this. Looks like it's due to BandedMatrices.jl which is a dep of BoundaryValueDiffEq.jl https://github.com/JuliaLang/julia/issues/24383

[ChrisRackauckas]

The hotfix is released