search

JuliaPlots / PlotlyJS.jl

Julia library for plotting with plotly.js
Other
413 stars 77 forks source link

Kaleido process in package initialisation hangs #473

Closed jd-foster closed 4 months ago

jd-foster commented 8 months ago

Described: https://discourse.julialang.org/t/plotlyjs-0-18-11-hangs-when-trying-using-but-only-the-first-time/105636

I am experiencing a strange behavior with PlotlyJS 0.18.11. The first time I try “using” the package, Julia hangs. If I interrupt the process (CTRL-C, I am on Windows 10) I get error different error messages in VSCode or in the REPL launched from a command prompt (see below). But in both cases, after interrupting the process, a second attempt to “using” works, and I haven’t noticed any problems with plots. I am on Julia 1.9.3, Windows 10, and just downgrading to PlotlyJS 0.18.10 fixes the issue in the environments I tried.

Version info Julia 1.9.3, Windows 10, and happens on upgrading PlotlyJS 0.18.10 to PlotlyJS 0.18.11.

Appears to be related to the recent change in #469 (possibly evident more often on Windows?).

empet commented 8 months ago

I have Julia 1.9.0, Windows 10, and PlotlyJS 0.18.11 works fine. I suspect that there is an incompatibility with Julia 1.9.3.

jd-foster commented 8 months ago

Thanks for the information. I can't reproduce this myself under the same versions, but the original poster of the issue can reproduce the hanging behaviour reliably inside VS Code & julia extension, but not so reliably from command-line REPL use.

BeastyBlacksmith commented 8 months ago

that might be fixed by replacing the kaleido setup here with https://github.com/JuliaPlots/PlotlyKaleido.jl

cserteGT3 commented 8 months ago

I also have similar issue. Sometimes I can load PlotlyJS, but what never works is Dash.jl+PlotlyJS (the extension precompilation always hangs). I'm on Windows 11, Julia v1.9.3, Dash v1.4.0 and PlotlyJS v0.18.11. I am happy to test any code that helps debugging. (I can use the 0.18.10 version until the julia PR mentioned in the discourse thread is merged and released.)

jd-foster commented 8 months ago

@cserteGT3 Thanks for reporting this. Do you have any steps you can share that reproduces this, even "sometimes"? If you observe those times that PlotlyJS loads, what does PlotlyJS._kaleido_running() give you?

cserteGT3 commented 8 months ago

I have the following environment:

(dashtemp) pkg> st
Status `C:\Users\username\dashtemp\Project.toml`
  [1b08a953] Dash v1.4.0
  [f0f68f2c] PlotlyJS v0.18.11

If I start a new julia session, the following happens (just updated to julia v1.9.4, therefore the recompiling and also possibly this was todays first call of PlotlyJS):

julia> using PlotlyJS
[ Info: Precompiling PlotlyJS [f0f68f2c-4968-5e81-91da-67840de0976a]

julia> PlotlyJS._kaleido_running()
true

julia> using Dash
[ Info: Precompiling Dash [1b08a953-4be3-4667-9a23-3db579824955]
[ Info: Precompiling DashBasePlotlyJSExt [1bcbdc22-5094-5a35-94fb-8d851f9434fe]

And the last line hangs. If I exit and start a new session, then using PlotlyJS in itself hangs. Sending ctrl+c will cause the following:

julia> using PlotlyJS
# pressing ctrl+c after a few minutes
┌ Warning: Kaleido is not available on this system. Julia will be unable to save images of any plots.
└ @ PlotlyJS C:\Users\username\.julia\packages\PlotlyJS\qhYQ5\src\kaleido.jl:65
┌ Warning: InterruptException()
└ @ PlotlyJS C:\Users\username\.julia\packages\PlotlyJS\qhYQ5\src\kaleido.jl:66

julia> PlotlyJS._kaleido_running()
true

If I remove the C:\Users\username\.julia\compiled\v1.9\PlotlyJS folder (thus forcing PlotlyJS to recompile next time) and start a fresh session using PlotlyJS hangs:

julia> using PlotlyJS
[ Info: Precompiling PlotlyJS [f0f68f2c-4968-5e81-91da-67840de0976a]
# pressing ctrl+c after a few minutes
┌ Warning: Kaleido is not available on this system. Julia will be unable to save images of any plots.
└ @ PlotlyJS C:\Users\csert\.julia\packages\PlotlyJS\qhYQ5\src\kaleido.jl:65
┌ Warning: InterruptException()
└ @ PlotlyJS C:\Users\csert\.julia\packages\PlotlyJS\qhYQ5\src\kaleido.jl:66

julia> PlotlyJS._kaleido_running()
true

The same happens if:

Then I removed the whole compiled folder again and killed the command line I was using, and started a new command line and a new julia session. using PlotlyJS still hangs as above.

I guess I should restart Windows, but can't do that now. Later I will try, if the first use of PlotlyJS works after restarting consistently.

I hope that this is useful in some way.

jd-foster commented 8 months ago

Thanks for all the extra detail.

disberd commented 8 months ago

that might be fixed by replacing the kaleido setup here with JuliaPlots/PlotlyKaleido.jl

I have to say that I experienced this issue (on Windows only) also when using PlotlyKaleido by itself, so it might not be fully solved by switching to PlotlyKaleido (which I believe is a worthy change in any case as it now has more features than the baked-in kaleido we have here)

make296 commented 8 months ago

Downgrading Julia to 1.5.4 solves the problem. Now can save plot to static image.

jd-foster commented 8 months ago

Downgrading Julia to 1.5.4 solves the problem.

@make296 I don't think that is a good idea to change your Julia version, especially anything less that the LTS version 1.6 . You're better off downgrading the package itself in most cases to the last version that worked for you.

make296 commented 8 months ago

Downgrading Julia to 1.5.4 solves the problem.

@make296 I don't think that is a good idea to change your Julia version, especially anything less that the LTS version 1.6 . You're better off downgrading the package itself in most cases to the last version that worked for you.

I reverted back to Julia 1.9.4 and followed step 2 in https://github.com/sglyon/PlotlyBase.jl/issues/40#issuecomment-774164216. It worked for me.

mattcbro commented 7 months ago

Well I can verify that importing PlotlyJS hangs on my windows laptop running Julia 1.9.4 within VScode. On Linux Mint it's problem free. On Linux the Julia libraries all load very quickly in general, whereas windows lags for some reason.

Hmm. I tried downgrading to PlotlyJS 0.18.10 and my code still hangs every time calling PlotlyJS.savefig(). So the downgrading PlotlyJS does not seem to be working for me.

mattcbro commented 7 months ago

Downgrading Julia to 1.5.4 solves the problem.

@make296 I don't think that is a good idea to change your Julia version, especially anything less that the LTS version 1.6 . You're better off downgrading the package itself in most cases to the last version that worked for you.

I reverted back to Julia 1.9.4 and followed step 2 in sglyon/PlotlyBase.jl#40 (comment). It worked for me.

This didn't work for me. Maybe I did things in the wrong order. Does PlotlyJS have to be installed first?

disberd commented 7 months ago

@mattcbro did you try forcing Kaleido_jll to v0.1.0? I found in my experiments with PlutoPlotly that it seems to fix the hanging problem on windows for me (and doesn't break saving plots from my short tests).

make296 commented 7 months ago

@mattcbro yes, install PlotlyJS and then replace Kaleido

mattcbro commented 7 months ago

@mattcbro yes, install PlotlyJS and then replace Kaleido

Whoops, did it in the wrong order. But it doesn't matter now. I install Linux as a windows subsystem to get a side by side Ubuntu distro with Win 11, and then installing julia and PlotlyJS allowed everything to work using the latest versions. It's curious that the linux version just works and MS windows, not so much.

beorostica commented 6 months ago

I created a small repository specifically for testing this issue, and it seems that the problem can be replicated in the GitHub CI. In that environment, I added PlotlyJS 0.18.11 in a subdirectory, allowing me to log some messages.

You can find all the experiments and output messages in the GitHub Actions at https://github.com/beorostica/TestPlotlyJS.jl/pull/2. For example, refer to https://github.com/beorostica/TestPlotlyJS.jl/actions/runs/7357215267/job/20028500523.

The _start_kaleido_process function gets stuck when executing:

...
res = readline(P.stdout)    # here hangs
...

I don't know how to address the problem from here, but if someone has any pointers, I would be happy to help 🙂.

cncastillo commented 6 months ago

Just want to add that what @beorostica describes is very similar to https://github.com/plotly/Kaleido/issues/134#issuecomment-1602468537. Maybe it helps.

disberd commented 4 months ago

This should now be solved after https://github.com/JuliaPlots/PlotlyKaleido.jl/pull/17 and https://github.com/JuliaPlots/PlotlyKaleido.jl/pull/18.

The kaleido process should not hang anymore but the kaleido library itself will hang in many windows machines for versions of Kaleido_jll > 0.1. This is unfortunately a problem that has nothing to do with julia but that is caused by the C++ library that is simply called by julia. It is also a problem that will likely not be solved soon as it's an open issue in https://github.com/plotly/Kaleido since years.

What https://github.com/JuliaPlots/PlotlyKaleido.jl/pull/17 did is simply wait for a timeout and kill the process if it appears to be hanging. It also gives an hopefully more informative warning with a suggested fix when this happens.

The simple fix is to force Kaleido_jll to version 0.1 on machines where this hanging occur (only windows and not all of windows machines). There was another issue causing problems with Windows and Kaleido only in julia 1.10 but that was also fixed in https://github.com/JuliaPlots/PlotlyKaleido.jl/pull/17 so now downgrading Kaleido_jll to 0.1 should effectively solve the problem on windows and allow to save figures.

I did not check in depth what downgrades in saving figures functionality using version 0.1 of the kaleido library brings compared to version 0.2, but all figures in my tests/use cases are fine with version 0.1

NOTE:

The hanging is effectively solved only while using version 2.2.4 of the PlotlyKaleido dependency, and only on PlotlyJS versions which actually use PlotlyKaleido as dependency (which is minimum version 0.18.3 of PlotlyJS). This should anyhow be what happens automatically if you add PlotlyJS in an environment from now on.

Due to this, I am now closing this issue as solved. We can re-open if this problem seems to persist for someone using the latest versions of the packages as explained above.

BeastyBlacksmith commented 4 months ago

Seems like calling PlotlyKaleido.start in __init__ is still causing the precompilation hang in julia 1.10.2 when using PlotlyJS in a package extension.

E.g. on the branch for Plots v2:

using Plots
Precompiling Plots
  79 dependencies successfully precompiled in 40 seconds. 44 already precompiled.
Precompiling PlotsPlotlyJSExt
        Info Given PlotsPlotlyJSExt was explicitly requested, output will be shown live 

[pid 52547] waiting for IO to finish:
 Handle type        uv_handle_t->data
 pipe[17]           0x312cde0->0x7f1f15246860
 pipe[11]           0x310bdc0->0x7f1f15246900
 process[52621]     0x3109f40->0x7f1f1524ff10
 timer              0x3046560->0x7f1f150f0e50
This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
disberd commented 4 months ago

@BeastyBlacksmith could this be related to Plots being only compatible with PlotlyKaleido@v1 based on the compat in the Project.toml of Plots?

BeastyBlacksmith commented 4 months ago

On https://github.com/JuliaPlots/Plots.jl/pull/4866 I set the compat to 2.2.2. Maybe it would make sense to start kaeleido in init to check if it works and kill it right after and then restart it when calling savefig?

disberd commented 4 months ago

On https://github.com/JuliaPlots/Plots.jl/pull/4866 I set the compat to 2.2.2. Maybe it would make sense to start kaeleido in init to check if it works and kill it right after and then restart it when calling savefig?

I assumed we were calling start in init to save the time to start the process, which won't happen if we also kill it in init. If it's just for precompilaton why don't we use PrecompileTools instead of init?

It would be cleaner that way

BeastyBlacksmith commented 4 months ago

To me that is a side benefit and its not about precompiling it. To me its about being able to warn the user something is off before you do any work, that you possibly can't save at the end.

The quest is to do that in a way that does not precompilation of the package itself and of dependent packages.

disberd commented 4 months ago

Ok I see, the easy way would be to just start and stop the kaleido process inside of init, but the strange thing is that start shouldn't be called during precompilation already because of this if: https://github.com/JuliaPlots/PlotlyJS.jl/blob/75eacf4542dc12d7b05242b550f250e6039929bf/src/PlotlyJS.jl#L102-L105

BeastyBlacksmith commented 4 months ago

In theory that shold work. Maybe that function returns 1 when precompiling extensions?