GR backend - Githubissues

tbreloff commented 8 years ago

Current TODO list (assume y = rand(10) for the examples):

[ ] Support the grid keyword. This should display a plot without the grid lines: plot(y, grid=false)
[ ] Support handling of NaN (gap in line segment): plot(vcat(1:10, NaN, 12:21), vcat(y,NaN,y))
[ ] Support removal/setting of xticks/yticks: plot(y, xticks=nothing, yticks=nothing)
[ ] Support xlim/ylim

tbreloff commented 8 years ago

I did some very basic initial tests. Here's some things that I consider bugs (but may be unavoidable, as I don't know GR very well):

only one plot window can be open at a time
if you close the window, there is a fatal error and GR is unusable for the rest of the session

In addition, there are tons of errors on the examples... I recommend doing the following:

Pkg.clone("https://github.com/tbreloff/ExamplePlots.jl.git")
using ExamplePlots
test_examples(:gr)

Note that test_examples also accepts an optional second argument which is the number of a specific example.

tbreloff commented 8 years ago

ps - this is the error when you close the window:

julia> XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":0"
      after 5740 requests (5585 known processed) with 0 events remaining.

jheinen commented 8 years ago

Thanks for merging the PR and doing initial tests. In terms of those results I would like to mention that:

GR can handle multiple (and different) output devices (simultaneously). I did not implement this feature in the initial release.
Closing the window (on Linux) will not re-open the X11 display. For applications that produce "figures" (like Plots), this behavior is inconvenient. I will provide a solution in the next GR release.
I will try to add the missing features (plot types) ASAP

BTW: It seems to me that some of the "bugs" are simply ignored by other backends, e.g.

test_examples(:gr,4)

After analyzing the debug output It turns out, that the markersize is given as an array:

Updating plot items:
...
    markersize: 100-element Array{Float64,1}
...

~~Does this make sense? PyPlot seems to ignore this.~~

In the meantime I noticed, that Plots tries to draw marker symbols with different colors and sizes.

tbreloff commented 8 years ago

Thanks for the thorough respone @jheinen. I think this contribution is great, and the time-to-plot seems very quick based on my very brief tests.

BTW: It seems to me that some of the "bugs" are simply ignored by other backends

Yes, Plots is intended to be a superset of functionality of the underlying backends. This means that a user should be able to do anything they want, but they may have to switch backends to accomplish it. Ideally it will be a little easier to understand exactly which features are supported by the backends... Much of this is recorded within supported.jl, but it's not at all complete. You found a good example with PyPlot's markersize... a scalar value is supported, but not an array. I think at some point, a re-write of the "supported" logic is in order. This has been on my radar for a while... see: #7

I'm not at all asking that you fully support all the examples... just that they will help you test your code, and maybe point you to missing functionality.

Thanks again for the contribution!

lobingera commented 8 years ago

two small things,

can antialiasing be turned on?
how to resize the window?

jheinen commented 8 years ago

Added TODO messages (see https://github.com/jheinen/Plots.jl/commit/c5d21813888bfcf697b81f5f308f69b12287be07) for missing features - I keep trying and don't expect any problems, except for animations, which are handled internally in GR.

jheinen commented 8 years ago

two small things,

can antialiasing be turned on? how to resize the window?

Anti-aliasing in GR currently only works for the Qt4 and (OS X) Quartz drivers. The next GR release will contain a cairographics driver which will add this feature in UN*X environments. Resizing, subplots and other things are still on the TODO list for the Plots backend.

lobingera commented 8 years ago

another small thing:

How to time()

plot(rand(8000000),rand(8000000))

runs a few seconds (wall clock), but

julia> tic(); plot(rand(800000),rand(800000)); toc()
elapsed time: 0.013522369 seconds
0.013522369

without any visible action on screen?

tbreloff commented 8 years ago

Andreas, this question is likely much more complicated and far reaching than Plots/GR. I suspect it has more to do with the time macro, and when the command is considered "complete". I've seen this in the past, and assumed it has more to do with the asynchronous event architecture underlying Julia (which I admit I only have partial knowledge of). Essentially julia hands off responsibility of the command to another thread and doesn't monitor completion of the task on that thread... It just marks it as "complete" when the handoff occurs. (This is mostly guessing of course)

On Jan 5, 2016, at 12:20 PM, Andreas Lobinger notifications@github.com wrote:

another small thing:

How to time() plot(rand(8000000),rand(8000000)) runs a few seconds (wall clock), but

julia> tic(); plot(rand(800000),rand(800000)); toc() elapsed time: 0.013522369 seconds 0.013522369 without any visible action on screen?

— Reply to this email directly or view it on GitHub.

KristofferC commented 8 years ago

Use tic(); display(plot(rand(800000),rand(800000))); toc()

Your code doesn't plot for the same reason that plot(rand(800000),rand(800000)); doesn't plot, (note the ;).

tbreloff commented 8 years ago

@KristofferC you're right I misread that... display is suppressed with a semicolon at the REPL (and always suppressed within scripts), so you need an explicit call to display (or gui(), which ensures it goes to the PlotsDisplay if you have others set).

The behavior I'm referring to is that at some part of the display pipeline (depending on the backend), julia will "hand off" processing, and will claim that the command is "complete", even though the gui window, browser window, or whatever else may not be complete. I just tested this with Gadfly, and the image was visibly loading, even though julia had already printed the toc() result.

KristofferC commented 8 years ago

Yeah, I guess some backends return and finish the drawing asynchronously.

jheinen commented 8 years ago

@tbreloff : Making progress but still some TODOs:

linetypes :~~path3d, :scatter3d, :surface, :wireframe, :ohlc,~~ :pie
~~colored contour lines~~
multiple figure output

http://pgi-jcns.fz-juelich.de/pub/doc/ExamplePlots.html

tbreloff commented 8 years ago

Looking good! Let me know when you're ready to try a merge.

On Mon, Jan 25, 2016 at 11:53 AM, Josef Heinen notifications@github.com wrote:

@tbreloff https://github.com/tbreloff : Making progress but still some TODOs:

linetypes :path3d, :scatter3d, :surface, :wireframe, :ohlc, :pie

colored contour lines

multiple figure output

http://pgi-jcns.fz-juelich.de/pub/doc/ExamplePlots.html

— Reply to this email directly or view it on GitHub https://github.com/tbreloff/Plots.jl/issues/108#issuecomment-174579525.

Evizero commented 8 years ago

This looks nice!

tbreloff commented 8 years ago

So that it doesn't get lost, I need to help with this request from @jheinen

@tbreloff : Need some help with the show / display logic. gr.jl has two writemime methods and both (PNG + SVG) are called when a plot is finalized (results in two files gks.png and gks.svg). Can I set the defaultOutputFormat to something other than PNG, e.g. SVG? The PNG writemime method should only be invoked when the user calls savefig with a .png file extension.

jheinen commented 8 years ago

Most of the stuff should work right now: http://pgi-jcns.fz-juelich.de/pub/doc/ExamplePlots.html

To create animation, I had to disable transparency when converting the PNGs to GIF (see commit 8cd6b0b). Without the -alpha off option, all images are overlayed. The change should not affect other backends.

Should I provide the output for the http://plots.readthedocs.org/en/latest/examples/gr/ section? I'd still prefer SVG :-) - it's your choice ...

tbreloff commented 8 years ago

These examples look really great. Nice work!

As for the doc examples, they are auto-generated with a script in ExamplePlots.jl. I have had the same thought about changing the format... For example Plotly would be best as html, since then you could see the interactivity. It's on my list and I'm open to PRs there if you have good ideas.

On Feb 13, 2016, at 3:45 AM, Josef Heinen notifications@github.com wrote:

Most of the stuff should work right now: http://pgi-jcns.fz-juelich.de/pub/doc/ExamplePlots.html

To create animation, I had to disable transparency when converting the PNGs to GIF (see commit 8cd6b0b). Without the -alpha off option, all images are overlayed. The change should not affect other backends.

Should I provide the output for the http://plots.readthedocs.org/en/latest/examples/gr/ section? I'd still prefer SVG :-) - it's your choice ...

— Reply to this email directly or view it on GitHub.

jheinen commented 8 years ago

Can I help to provide the doc examples for GR? Would it help to force PNG for the inline format, e.g. thru a configuration option or an environment variable (PLOTSBACKEND, like MPLBACKEND for matplotlib)?

tbreloff commented 8 years ago

Yes you might be able to help. The examples are generated from ExamplePlots, using this method: https://github.com/tbreloff/ExamplePlots.jl/blob/master/src/example_generation.jl#L206

I tried running it and it crashed for me... Maybe you'll be able to debug? If you get it working a PR would be great.

On Feb 27, 2016, at 6:03 AM, Josef Heinen notifications@github.com wrote:

Can I help to provide the doc examples for GR? Would it help to force PNG for the inline format, e.g. thru a configuration option or an environment variable (PLOTSBACKEND, like MPLBACKEND for matplotlib)?

— Reply to this email directly or view it on GitHub.

jheinen commented 8 years ago

Added missing marker shapes (GR native) ...

tbreloff commented 8 years ago

It seems like you've made a lot of progress on your fork of Plots... Any interest in submitting a PR to my dev branch soon?

On Mar 5, 2016, at 7:01 AM, Josef Heinen notifications@github.com wrote:

Added missing marker shapes (GR native) ...

— Reply to this email directly or view it on GitHub.

jheinen commented 8 years ago

This patch requires a new GR build, which is already finished for the UN*X systems. But currently I don't have access to our Windows build system - so, next week, when I'm back in the office, I'll make a PR.

tbreloff commented 8 years ago

Sounds good. Thanks.

On Mar 5, 2016, at 8:19 AM, Josef Heinen notifications@github.com wrote:

This patch requires a new GR build, which is already finished for the UN*X systems. But currently I don't have access to our Windows build system - so, next week, when I'm back in the office, I'll make a PR.

— Reply to this email directly or view it on GitHub.

jheinen commented 8 years ago

Still don't understand, why both mime methods (svg + png) are called.

Nevertheless, make a PR now ...

tbreloff commented 8 years ago

First: Will it be possible to have custom marker shapes?

NOTE: the first time I run this, I get the error:

KeyError: Plots.Shape([(-1,-1),(1,0),(-1,1)]) not found

and the next time I run this I get the error:

attempt to save state twice KeyError: Plots.Shape([(-1,-1),(1,0),(-1,1)]) not found in gr_display at /home/tom/.julia/v0.4/Plots/src/backends/gr.jl:316 in writemime at /home/tom/.julia/v0.4/Plots/src/backends/gr.jl:682 in sprint at iostream.jl:206 in display_dict at /home/tom/.julia/v0.4/IJulia/src/execute_request.jl:29

~~As for the second error... I wonder if a pattern of try/catch/finally would be a better solution:~~ Solved with @gr_state macro

julia> for i=1:10
           try
               println("save")
               rand()<0.3 && error()
           catch err
               println(err)
           finally
               println("restore")
           end
       end
save
restore
save
restore
save
restore
save
restore
save
ErrorException("")
restore
save
ErrorException("")
restore
save
restore
save
restore
save
ErrorException("")
restore
save
restore

We could even add a macro to do this:

@state begin
    # do something
end

which could expand to:

try
  GR.savestate()
  # code block
catch err
  # do something with err
finally
  GR.restorestate()
end

At a minimum, this would clean up the code significantly. Thoughts?

tbreloff commented 8 years ago

I added this macro (easier to do than discuss):

macro gr_state(expr::Expr)
  esc(quote
    GR.savestate()
    try
      $expr
    catch
      GR.restorestate()
      rethrow()
    end
    GR.restorestate()
  end)
end

and the state error goes away. (obviously the Shape error is still there, but at least it doesn't corrupt the session)

tbreloff commented 8 years ago

~~This drives me nuts... I'd really like to figure out a solution:~~ Solved by clearing/rebuilding GR


julia> using Plots; gr()
Plots.GRPackage()

julia> plot(rand(10))
[Plots.jl] Initializing backend: gr

     [ I manually close the GUI window... ]

julia> XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":0"
      after 284 requests (195 known processed) with 0 events remaining.

julia> plot(rand(10))

    [ Hangs forever... ]

Ideally there would be cleanup that happens when the window is closed so that the session is not ruined.

tbreloff commented 8 years ago

For the state macro, see: https://github.com/tbreloff/Plots.jl/commit/cdff5a9039c2f51b041a819cbdfc2dc72ba67ac9

jheinen commented 8 years ago

It seems that your are not using the GR master branch. The XIO problems has been solved in the meantime. Please rm, free and add the GR package.

I'd prefer to "officially" support the Shape function instead of catching exceptions. What should Plots.Shape([(-1,-1),(1,0),(-1,1)]) do? Does it simply create two lines given by three points?

jheinen commented 8 years ago

Pls keep in mind: the GR build.jl script will not download the run-time if it finds a locally installed GR, e.g. in $HOME/gr, /usr/local/gr or as specified by the GRDIR environment.

KristofferC commented 8 years ago

Can confirm that I have no problems closing and opening new plots with the lastest GR.

tbreloff commented 8 years ago

your are not using the GR master branch

Ok reinstalling GR now.

I'd prefer to "officially" support the Shape function instead of catching exceptions.

Of course we want everything to work, but that doesn't change the need for graceful handling of errors.

What should Plots.Shape([(-1,-1),(1,0),(-1,1)]) do?

It should define a marker shape (in "unit coordinates centered at zero"). Try in another backend (for example, gadfly):

shape = Shape([(-1,-1),(1,0),(-1,1)])
plot(rand(10), marker = (shape, 20, 0.3))

tbreloff commented 8 years ago

After manually clearing all traces of GR from my ~/.julia directory, I was able to rebuild GR, and I confirm the window-closing-error is gone. Hurrah!

tbreloff commented 8 years ago

I have a temporary fix for the Shape issue here: https://github.com/tbreloff/Plots.jl/commit/9d3e0651e2848144a51a47779b983d204d59c306

tbreloff commented 8 years ago

I updated the first comment with a TODO list... I'll try to add/remove as we make progress.

Thanks again for working on this @jheinen... I'm looking forward to using GR for my work.

jheinen commented 8 years ago

... I'm getting closer (see e2e031c).

The TODOs should be easy to implement. But I'm getting sleepy ...

tbreloff commented 8 years ago

Just out of curiosity I added GR to the travis tests. The linux test passes, osx fails with:

[Plots.jl] Initializing backend: gr
INFO: Testing plot: gr:1:Lines
GKS: Ghostscript support not compiled in
  Error :: (line:505)
    Expression: image_comparison_tests(pkg,i,debug=debug,sigma=sigma,eps=eps) |> success --> true
    SystemError: opening file gks.png: No such file or directory
     in open at /Users/travis/julia/lib/julia/sys.dylib (repeats 2 times)
     in writemime at /Users/travis/.julia/v0.4/Plots/src/backends/gr.jl:693
     in png at /Users/travis/.julia/v0.4/Plots/src/output.jl:8
     in png at /Users/travis/.julia/v0.4/Plots/src/output.jl:11
     in anonymous at /Users/travis/.julia/v0.4/Plots/test/imgcomp.jl:40
     in test_images at /Users/travis/.julia/v0.4/VisualRegressionTests/src/imgcomp.jl:79
     in image_comparison_tests at /Users/travis/.julia/v0.4/Plots/test/imgcomp.jl:52
     in anonymous at /Users/travis/.julia/v0.4/FactCheck/src/FactCheck.jl:271
     in do_fact at /Users/travis/.julia/v0.4/FactCheck/src/FactCheck.jl:333
     [inlined code] from /Users/travis/.julia/v0.4/FactCheck/src/FactCheck.jl:271
     in image_comparison_facts at /Users/travis/.julia/v0.4/Plots/test/imgcomp.jl:61
     in anonymous at /Users/travis/.julia/v0.4/Plots/test/runtests.jl:33
     in facts at /Users/travis/.julia/v0.4/FactCheck/src/FactCheck.jl:448
     in include at /Users/travis/julia/lib/julia/sys.dylib
     in include_from_node1 at /Users/travis/julia/lib/julia/sys.dylib
     in process_options at /Users/travis/julia/lib/julia/sys.dylib
     in _start at /Users/travis/julia/lib/julia/sys.dylib
Out of 3 total facts:
  Verified: 2
  Errored:  1

jheinen commented 8 years ago

Can I access the travis.ci log to verify the build step?

tbreloff commented 8 years ago

This is what I see for GR:

INFO: Building GR

INFO: Downloading pre-compiled GR 0.17.3 binary

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0 20.3M    0  118k    0     0   120k      0  0:02:53 --:--:--  0:02:53  120k
  7 20.3M    7 1573k    0     0   768k      0  0:00:27  0:00:02  0:00:25  769k
 17 20.3M   17 3745k    0     0  1236k      0  0:00:16  0:00:03  0:00:13 1236k
 28 20.3M   28 6008k    0     0  1502k      0  0:00:13  0:00:03  0:00:10 1502k
 40 20.3M   40 8397k    0     0  1689k      0  0:00:12  0:00:04  0:00:08 1689k
 52 20.3M   52 10.7M    0     0  1848k      0  0:00:11  0:00:05  0:00:06 2189k
 65 20.3M   65 13.3M    0     0  1965k      0  0:00:10  0:00:06  0:00:04 2464k
 80 20.3M   80 16.4M    0     0  2096k      0  0:00:09  0:00:08  0:00:01 2618k
 94 20.3M   94 19.1M    0     0  2188k      0  0:00:09  0:00:08  0:00:01 2740k
100 20.3M  100 20.3M    0     0  2239k      0  0:00:09  0:00:09 --:--:-- 2867k

jheinen commented 8 years ago

Ok. I should tag a new GR version ASAP - the latest pre-compiled binary is required:

$ julia --color=yes -e "Pkg.build(\"${JL_PKG}\")"
INFO: Building GR
INFO: Downloading pre-compiled GR latest binary
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20.3M  100 20.3M    0     0  2860k      0  0:00:07  0:00:07 --:--:-- 3970k
19.00s$ julia --check-bounds=yes --color=yes -e "Pkg.test(\"${JL_PKG}\", coverage=true)"
INFO: Testing GR
running /Users/travis/.julia/v0.4/GR/test/ex.jl ...
running /Users/travis/.julia/v0.4/GR/test/griddata.jl ...
INFO: GR tests passed

tbreloff commented 8 years ago

Ok no hurry from my end... I was just curious whether it would pass.

tbreloff commented 8 years ago

Still don't understand, why both mime methods (svg + png) are called.

It seems this is intentional on the part of IJulia (seems like a bad decision to me, but I'm sure there's a reason for it):

https://github.com/JuliaLang/IJulia.jl/blob/master/src/inline.jl#L31-L40

I'm considering overriding this method for Plots so we're not constantly rebuilding multiple plots.

tbreloff commented 8 years ago

IJulia weirdness fixed on dev. See #157

tbreloff commented 8 years ago

I feel like it's really important we change the behavior of writing out a temporary file gks.png to the current working directory. If you need a temporary file, create one with tempname() * ".png". It drives me insane to have every directory polluted with gks.* files.

tbreloff commented 8 years ago

For the broken markers example:

I see this error when running from the REPL:

julia> scatter(x,y,m=(8,:auto),lab=map(string,markers),bg=:linen,xlim=(0,10),ylim=(0,10))
GKS: unable to load font

It might be unrelated, but figured I'd point it out.

tbreloff commented 8 years ago

To summarize what I see for the markers example:

It works fine if you run it from the REPL and display in the window.
When you call png(...) it returns as if successful, but the file is whatever was last saved.

Presumably there is a silent error somewhere (probably inside GR.jl? I can't find any try blocks in the Plots code), only when displaying to a png, and so the gks.png file is never written to.

Do you have any ideas of how to investigate?

tbreloff commented 8 years ago

Question... does GR support alpha values for colors? (either RGBA or setting the alpha seperately?)

jheinen commented 8 years ago

@tbreloff : I see the same behavior for the marker example (except the GKS: unable to load font error). When I remove the :auto parameter, it works as expected. If :auto is added to the marker options, the gr() methods are not called.

I still have no idea, but there is something going wrong: When I use the pyplot() backend I get the following output:

Using gks.png as the default output file (in the current directory) is not a good idea. I will change this ASAP.

jheinen commented 8 years ago

Question... does GR support alpha values for colors? (either RGBA or setting the alpha seperately?)

The alpha channel can be set with gr.settransparency(a), 0 <= a <=1.

JuliaPlots / Plots.jl

GR backend #108