DifferentialEquations precompiles on every launch

[RaimundK]

While Pluto has many nice features and may be a great development environment, its precompilation is really annoying as soon as more than 1-2 packages are used. I have a simple notebook, starting with begin using DifferentialEquations using PlutoUI, Plots using LinearAlgebra using CSV, DataFrames end (-> some calculations (defining and solving some ODES) afterwards)

At the first time, when building up the notebook, precompilation phase was around 20 min, afterwards I was able to calculate in reasonable time. Afterwards, each time when I opened the file, precompilation takes still around 10 minutes and afterwards I stopped calculation after around 15 minutes without any result (it seemed that still only one cell was evaluated) Needless to say that the same usings in the REPL or in jupyterlab are much faster each time ...

[ederag]

Confirmed with Pluto v0.19.40 and julia 1.10.2.

Paste and choose "split"

using DifferentialEquations
using PlutoUI, Plots
using LinearAlgebra
using CSV, DataFrames

Took 392 s to precompile, then several minutes "Evaluating cells", then I tried to add using Pkg to check a few things, and it started a lengthy precompile again:

Then I closed the notebook, reopened, "Package Management" was fast, but it took several minutes "Evaluating cells" this time.

Then closed the notebook, reopened, and it took less than 10s to be ready.

[RaimundK]

The problem is that such things happen, if there are more than 1-2 packages loaded, which makes unfortunately (!) Pluto hard to use for any slightly complex tasks, at least currently

[PhyX-Meow]

I have the same issue. I use the global package environment from import Pkg; Pkg.activate(); using .... But it seems pluto failed to recognize the cache julia created, but does another precompilation. When pluto's precompilation completed, I start a julia REPL and using the packages, then I need another precompilation. There has to be something wrong.

Julia v1.10.2, Pluto v0.19.40

[RaimundK]

The same using statements on jupyterlab (in a second session after activating the environment and adding the packages in afirst session) take ~20 seconds in each following session.

[PhyX-Meow]

This is somehow fixed for me, after several package updates. Don't know exactly why...

[fonsp]

Which Julia version do you use? @RaimundK

[fonsp]

@ederag Could you elaborate a bit with what you meant in your comment? What is the issue? What should we fix? Do you see something different in the REPL? If you're really precise then there is something we can fix!

@RaimundK @PhyX-Meow It would be really helpful to get a reproducing example: this is a list of steps that anyone (like me) can follow to see the problem that you saw. E.g. open this notebook file (link to a github gist) with Julia v1.2.3 and Pluto v1.2.3, run, shut down, etc etc. A video recording is also helpful, see instructions.

So instead of reporting what you saw, try to give me instructions so that I can see it myself. Only then can we fix it :)

[ederag]

@fonsp

Could you elaborate a bit with what you meant in your comment? What is the issue? What should we fix? Do you see something different in the REPL?

https://github.com/fonsp/Pluto.jl/issues/2846#issuecomment-2001924196 was just aimed at giving steps to reproduce the behavior in the OP, without any speculation.

REPL

Today I tried in the REPL first, and it was the same (few minutes to precompile DifferentialEquations mostly, which is understandable), until using Pkg, which was fast in the REPL.

Note: a second attempt in a fresh julia session took few minutes again, but this can be understood as in the meantime two packages were updated in the registry, which asked for recompilation of the whole.

A third attempt in a fresh julia session was fast. In all three attempts, using Pkg was instant.

Pluto

Then trying in Pluto, with the same procedure, took several minutes, although no updates were visible in the "Pkg terminal output".

Closing the notebook and reopening it was fast. using Pkg was fast (it does launch a precompilation, but a quick one).

During these operations, there was no output in the terminal from where Pluto was launched.

My current interpretation would be that each of those minutes-long recompilations mentioned above are legit, triggered by packages updates in the general repository, and not specific to Pluto ?

[RaimundK]

@ederag No, even after going back to the main page, stopping the notebook and then opening and running it again leads to the same long times. No packes habve been updated meanwhile. The problem does not exist, when using e.g. jupyterlab, after the first compilation, subsequent usage is fast.

@fonsp Open any new notebook, paste the lines begin using DifferentialEquations using PlutoUI, Plots using LinearAlgebra using CSV, DataFrames end in a cell add a second cell: 1+1 start after precompilation was successful, close the notebook, stop the process reopen the notebook and run it close again and stop open again (observe the times. in particular when opening the 2nd, 3d etc time, try the same process with several other packages) ...

Sorry, but a film would be very long and boring ...

[fonsp]

Can you share an HTML export or a notebook file (so that the version numbers are pinned) and share the 1st, 2nd and 3rd times that you measured? A screenshot of Pluto's Status tab (bottom right) with all items expanded after the 1st, 2nd and 3rd runs would be very helpful. Also send us your Julia and Pluto version numbers.

Which of these packages is causing the issue? Can you try to eliminate and make your example smaller?

A "long and boring" video would also be very helpful! Put it on youtube if it's too long for github.

Remember that Pluto is free and open source, we really rely on contributions from the community to do what we do. If you can spare some time to give us the information we need to fix a problem then we are happy to do it. We are asking for a small effort on your part, compared to a large effort that we put into making Pluto and fixing issues.

[RaimundK]

@fonsp Thans for your efforts!

Please find attached some information about a similar case. Not exactly the packages used above, but similar issues - took long time for precompiling then even much more time before even evaluating something. 1) notebook 2) screenshot of durations 3) screenshot of errors in the julia console, happening after some time during the whole process ...

Julia version: Julia Version 1.10.2 Commit bd47eca2c8 (2024-03-01 10:14 UTC) Build Info: Official https://julialang.org/ release Platform Info: OS: Windows (x86_64-w64-mingw32) CPU: 8 × Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-15.0.7 (ORCJIT, skylake) Threads: 4 default, 0 interactive, 2 GC (on 8 virtual cores)

Environment: JULIA_EDITOR = mate JULIA_NUM_THREADS = 4

Pluto Version: Pluto v0.19.40

https://www.dropbox.com/scl/fo/mc4pt2iegjsaqbylj0czx/h?rlkey=07enyy82q89590gwo2ovo92ug&dl=0

[fonsp]

Your problem is that you run the same notebook 3 times, and 3 times it takes a long time to run. Can you:

1. Run the notebook once, export as run1.html. Take a screenshot of the status tab: status1.png.
2. Shut down and run again, export as run2.html. Take a screenshot of the status tab: status2.png.
3. Shut down and run again, export as run3.html. Take a screenshot of the status tab: status3.png.

Send me those 6 files.

How to export as HTML

https://github.com/fonsp/Pluto.jl/assets/6933510/ab2cdaa8-d10f-4ad8-9539-9a67501d46bb

[fonsp]

I can't reproduce. I suspect that there was an issue with (one of the deps of) DiffEq that caused precompilation to be triggered every time. I'll close this until I have some actionable information