multiple calls to annotation_custom fail in certain cases

[fmitha]

I'm copying the example from Sandy Musgrove in http://stackoverflow.com/a/13327793/350713

In the following code, two calls to annotation_custom result in the object plotNew. But the two legends that should show up in that plot don't, (We can call up the display by simply typing plotNew.) Only the first added shows up, namely leg2. However, in the case of plotNew2, both circles added via annotation_custom show up.

library(ggplot2)
library(gtable)
## The proto package needs to be loaded for more detailed printing of proto objects
## I assume this replaces the `print` method for proto objects
## Without this package loaded you don't see `str` errors below.
library(proto)

##Some data
df <- data.frame(
  x = 1:10,
  y = 1:10,
  colour = factor(sample(1:3, 10, replace = TRUE)),
  size = factor(sample(1:3, 10, replace = TRUE)))

### Step 1
# Draw a plot with the colour legend
(p1 <- ggplot(data = df, aes(x=x, y=y)) +
   geom_point(aes(colour = colour)) +
   theme_bw() +
   theme(legend.position = "top"))

## Extract the colour legend - leg1
leg1 <- gtable_filter(ggplot_gtable(ggplot_build(p1)), "guide-box") 

## Step 2
## Draw a plot with the size legend
(p2 <- ggplot(data = df, aes(x=x, y=y)) +
   geom_point(aes(size = size)) +
   theme_bw())

## Extract the size legend - leg2
leg2 <- gtable_filter(ggplot_gtable(ggplot_build(p2)), "guide-box") 

## Step 3
## Draw a plot with no legends - plot
(plot <- ggplot(data = df, aes(x=x, y=y)) +
   geom_point(aes(size = size, colour = colour)) +
   theme_bw() +
   theme(legend.position = "none"))

plotNew <- plot + 
  annotation_custom(grob = leg2, xmin = 7, xmax = 10, ymin = 0, ymax = 4) +
  annotation_custom(grob = leg1, xmin = 2.5, xmax = 5, ymin = 7.5, ymax = 10)

plotNew2 <- plot +
  annotation_custom(circleGrob()) + annotation_custom(circleGrob(x=0.2, y=0.2, r=0.2))

Tested with ggplot2 0.9.3.1 and gtable 0.1.2 on Debian squeeze.

[fmitha]

Something seems to go wrong with ggplot_build here. Define

pbuild = ggplot_build(plotNew)

Then,

 str(pbuild$plot$layers)

gives an error. The printout ends with

 $ :proto object 
Error in object[[i]] : subscript out of bounds

I haven't been able to isolate exactly where the problem is. It looks like the error is with one of components of the pbuild$plot$layers list, but printing them individually, i.e. str(pbuild$plot$layers[[i]]) for i=1, 2, 3, gives no errors.

[fmitha]

I think the problem is here.

> str(pbuild$plot$layers[[3]]$geom_params$grob$grobs[[1]]$grobs[[1]], max.level=0)
List of 11
 - attr(*, "class")= chr [1:3] "gtable" "grob" "gDesc"

grid.draw(pbuild$plot$layers[[3]]$geom_params$grob$grobs[[1]]$grobs[[1]])

shows that it is the legend.

We also get

> pbuild$plot$layers[[3]]$geom_params$grob$grobs[[1]]$grobs[[1]]
TableGrob (4 x 15) "layout": 11 grobs
    z         cells        name                             grob
1   1 ( 1- 4, 1-15)  background rect[legend.background.rect.550]
2   2 ( 2- 3, 2- 2)       title       text[guide.title.text.533]
3   3 ( 2- 2, 4- 4)  key-1-3-bg        rect[legend.key.rect.541]
4   4 ( 2- 2, 4- 4)   key-1-3-1          points[GRID.points.542]
5   5 ( 2- 2, 8- 8)  key-1-7-bg        rect[legend.key.rect.544]
6   6 ( 2- 2, 8- 8)   key-1-7-1          points[GRID.points.545]
7   7 ( 2- 2,12-12) key-1-11-bg        rect[legend.key.rect.547]
8   8 ( 2- 2,12-12)  key-1-11-1          points[GRID.points.548]
9   9 ( 2- 2, 6- 6)   label-1-5       text[guide.label.text.535]
10 10 ( 2- 2,10-10)   label-1-9       text[guide.label.text.537]
11 11 ( 2- 2,14-14)  label-1-13       text[guide.label.text.539]

but trying to access the 11th component gives:

> pbuild$plot$layers[[3]]$geom_params$grob$grobs[[1]]$grobs[[1]][[11]]
Error in pbuild$plot$layers[[3]]$geom_params$grob$grobs[[1]]$grobs[[1]][[11]] : 
  subscript out of bounds

So this object reports itself as a list of 11 components, but only has 10 components.

Also the above remarks apply to

pbuild$plot$layers[[2]]$geom_params$grob$grobs[[1]]$grobs[[1]]

which corresponds to the other legend.

[fmitha]

Ok, it looks like the problem is already present earlier. Let us call

plotNew <- plot + annotation_custom(grob = leg2, xmin = 7, xmax = 10, ymin = 0, ymax = 4)

where plot is defined as in STEP 3 in the first post.

Then

str(plotNew$layers[[2]]$geom_params$grob$grobs[[1]]$grobs[[1]])

gives an error, and as above

> str(plotNew$layers[[2]]$geom_params$grob$grobs[[1]]$grobs[[1]], max.level=0)
List of 11

but as before there is no 11th component.

> plotNew$layers[[2]]$geom_params$grb$grobs[[1]]$grobs[[1]][[11]]
Error in plotNew$layers[[2]]$geom_params$grob$grobs[[1]]$grobs[[1]][[11]] : 
  subscript out of bounds

NB: The proto package needs to be loaded, otherwise you don't see any of these errors.

[fmitha]

It looks like this problem is already in the annotation_custom component. Define

 A =  annotation_custom(grob = leg2, xmin = 7, xmax = 10, ymin = 0, ymax = 4)

Then

> str(A$geom_params$grob$grobs[[1]]$grobs[[1]], max.level=0)
List of 11
 - attr(*, "class")= chr [1:3] "gtable" "grob" "gDesc"

> A$geom_params$grob$grobs[[1]]$grobs[[1]][[11]]
Error in A$geom_params$grob$grobs[[1]]$grobs[[1]][[11]] : 
  subscript out of bounds

NOTE: This may unrelated to the topic of this issue, but is an annoying distraction in any case.

[fmitha]

This can be traced back to the original legend grob object leg2 extracted from the plot p2.

> str(leg2$grobs[[1]]$grobs[[1]], max.level=0)
List of 11
 - attr(*, "class")= chr [1:3] "gtable" "grob" "gDesc"

> leg2$grobs[[1]]$grobs[[1]][[11]]
Error in leg2$grobs[[1]]$grobs[[1]][[11]] : subscript out of bounds

The same apply to leg1, of course. Since gtable_filter does the extraction, maybe the error is there.

[fmitha]

It looks like this problem precedes gtable_filter. If we define

gp2 = ggplot_gtable(ggplot_build(p2))

Then calling either

str(gp2)

or

str(gp2$grobs[[8]]$grobs[[1]])

ends with $ NA:Error in object[[i]] : subscript out of bounds and as before

> str(gp2$grobs[[8]]$grobs[[1]], max.level=0)
List of 11
 - attr(*, "class")= chr [1:3] "gtable" "grob" "gDesc"
> gp2$grobs[[8]]$grobs[[1]][[11]]
Error in gp2$grobs[[8]]$grobs[[1]][[11]] : subscript out of bounds

[fmitha]

Narrowing this down a bit more. The culprit seems to be gtable_add_grob. It it is called in ggplot_gtable for the position == "right" case as follows:

  plot_table <- gtable_add_grob(plot_table, legend_box, clip = "off",
      t = panel_dim$t, b = panel_dim$b, l = -1, r = -1, name = "guide-box")

My debug strategy is to add print(str(plot_table$grobs)) whenever plot_table changes. The first of these invocations to crash with the "out of bounds" error is after the gtable_add_grob line above.

[fmitha]

I think I see where this happens. First, the call stack.

ggplot_gtable (in "plot-render.r")
-> build_guides here: 
    build_guides(plot$scales, plot$layers, plot$mapping, position, theme, plot$guides, plot$labels)

build_guides (in "guides-.r")
-> guides_gengrob here: ggrobs <- guides_gengrob(gdefs, theme)

guide_gengrob is a wrapper.

In "guides-.r" we have

guide_gengrob <- function(...) UseMethod("guide_gengrob")

so calls

guide_gengrob (in "guides-.r") -> guide_gengrob.legend (in
"guide-legend.r)

guide_gengrob.legend (in "guide-legend.r) -> gtable_add_grob here: gt <- gtable_add_grob(gt, grob.labels,...

In

gtable_add_grob (in "add-grob.r" (gtable package))

the problem happens with x after running the lines

x$grobs <- c(x$grobs, grobs)

and

x$layout <- rbind(x$layout, layout)

If we dump the x before these lines, as well as the corresponding grobs and layout, then one can reproduce this problem.

I've included details on reproduction, as well as a dump of x, grobs and layout here.

https://bitbucket.org/faheem/github-ggplot2-817

The current README for this is included below. Summary: loading ggplot2 makes str give an error for reasons that are not obvious.

#####################################################################

SUMMARY: loading the ggplot2 package produces errors with str which are not present otherwise.

To reproduce this bug, do the following.

Start R. Then

> ls()
character(0)
> load("addgrob.asc.save")
> ls()
[1] "grobs"  "layout" "x"

> str(x, max.level=1)
List of 10
 $ grobs   :List of 8
 $ layout  :'data.frame':       8 obs. of  7 variables:
   $ widths  :Class 'unit'  atomic [1:6] 1.5 6.096 0.762 1.961 0 ...
  .. ..- attr(*, "unit")= chr "mm"
  .. ..- attr(*, "valid.unit")= int 7
 $ heights :Class 'unit'  atomic [1:7] 1.5 2.53 1.52 6.1 6.1 ...
  .. ..- attr(*, "unit")= chr "mm"
  .. ..- attr(*, "valid.unit")= int 7
 $ respect : logi FALSE
 $ rownames: NULL
 $ colnames: NULL
 $ name    : chr "layout"
 $ gp      : NULL
 $ vp      : NULL
 - attr(*, "class")= chr [1:3] "gtable" "grob" "gDesc"

> x$grobs <- c(x$grobs, grobs)
> x$layout <- rbind(x$layout, layout)

> str(x, max.level=1)
List of 10
 $ grobs   :List of 11
 $ layout  :'data.frame':       11 obs. of  7 variables:
   $ widths  :Class 'unit'  atomic [1:6] 1.5 6.096 0.762 1.961 0 ...
  .. ..- attr(*, "unit")= chr "mm"
  .. ..- attr(*, "valid.unit")= int 7
 $ heights :Class 'unit'  atomic [1:7] 1.5 2.53 1.52 6.1 6.1 ...
  .. ..- attr(*, "unit")= chr "mm"
  .. ..- attr(*, "valid.unit")= int 7
 $ respect : logi FALSE
 $ rownames: NULL
 $ colnames: NULL
 $ name    : chr "layout"
 $ gp      : NULL
 $ vp      : NULL
 - attr(*, "class")= chr [1:3] "gtable" "grob" "gDesc"

> library(ggplot2)

> str(x, max.level=1)
List of 11
 $ grobs   :List of 11
 $ layout  :'data.frame':       11 obs. of  7 variables:
   $ widths  :Class 'unit'  atomic [1:6] 1.5 6.096 0.762 1.961 0 ...
  .. ..- attr(*, "unit")= chr "mm"
  .. ..- attr(*, "valid.unit")= int 7
 $ heights :Class 'unit'  atomic [1:7] 1.5 2.53 1.52 6.1 6.1 ...
  .. ..- attr(*, "unit")= chr "mm"
  .. ..- attr(*, "valid.unit")= int 7
 $ respect : logi FALSE
 $ rownames: NULL
 $ colnames: NULL
 $ name    : chr "layout"
 $ gp      : NULL
 $ vp      : NULL
 $ NA:Error in object[[i]] : subscript out of bounds

[fmitha]

I posted a Stack Overflow question: calling str on a object errors out if the ggplot2 package is loaded.

It looks unlikely that this is what is causing the problem that is the topic of this issue, but it does make debugging harder if str errors out.

[fmitha]

On the assumption that this is a different problem from the topic of this issue, I have now created a separate issue for it - https://github.com/hadley/ggplot2/issues/820

[baptiste]

regarding the original issue, we've rediscovered it on Stack Overflow: http://stackoverflow.com/a/17785797/471093

"Interestingly", what the grobs are seems to affect the behaviour of annotation_custom; for instance adding two ggplotGrob fails (only one appears), but wrapping them in a gTree works. Weird stuff.

[hadley]

This sounds like a great feature/horrible bug, but unfortunately we don't currently have the development bandwidth to support it/fix it. If you'd like to submit a pull request that implements this feature/fixes this bug, please follow the instructions in the development vignette.

[fmitha]

I don't understand why you are closing this issue when the bug is not fixed.

[BBrill]

I'm trying to reproduce the graphic you made for the ozone dataset (the one with star gliphs) for the plyr paper (fig.11). Evidently you do not used the annotation_grob, could you explain us with some code how you made this graph? Thank you.

[fmitha]

@BBrill : I'm not sure who you are addressing here. Please be more specific. If you are addressing me, I opened this issue a long time ago, and don't remember anything at the moment. I'd have to look at it to refresh my memory, and frankly I'm not that motivated to spend time on an issue the developers have closed without bothering to even look at.

[BBrill]

@fmitha sorry, I'm adressing to Hadley, I didn't state it directly, but I adressed to the writer of the paper about plyr appeared on the Journal of Statistical software...

[BBrill]

I have discovered the ggsubplot package, which is made for the kind of graphics I mentioned early. Thank you very much, for this work!