Add error bars to summarized frames

[jhofman]

we discussed adding error bars to the summarized plots.

@giorgi-ghviniashvili: can we do this with just one "layer" in vegalite, or do we have to hack things again? could you play with this in the vegalite editor and then see if gemini can handle it?

if it is possible (🤞), then @sharlagelfand, we'll just need to add to the vegalite spec when exporting.

[giorgi-ghviniashvili]

@jhofman error bars are available using layer field, which is not supported by gemini. Box plots does not use layers, but does all the quantile calculations internally.

[jhofman]

could we either a) fake them in the same way you're doing with facet axes, by creating a second layer or b) try to add to gemini internally as per #49?

[giorgi-ghviniashvili]

@jhofman which one should I try first? I have not started gemini code reading yet.

[jhofman]

a) seems easier if it's possible to just turn on an extra layer with the error bars

[giorgi-ghviniashvili]

@sharlagelfand can you prepare specs for this for me to work with?

[sharlagelfand]

@giorgi-ghviniashvili I'm having some trouble figuring out error bars with facets - have you done this before?

[giorgi-ghviniashvili]

@sharlagelfand no , I have never done error bars with facets before. I only have this example: https://vega.github.io/vega-lite/examples/layer_point_errorbar_stdev.html

[giorgi-ghviniashvili]

Some progress:

https://user-images.githubusercontent.com/6615532/118880230-b0126200-b902-11eb-91a5-156a8b10c9bf.mov

[sharlagelfand]

@giorgi-ghviniashvili Here is an example of having error bars in facets, and how it looks:

I realized the domain isn't really an issue because it's the full domain of the data, so.... of course the error bar should fit within that 😅

In the summary spec, we precalculate the mean (and each point has the exact same value) and then send that over to vega lite, but in order for this to work we need to send the original values themselves to be used for the error bars. So rather than switch the y values (in case that messes with the animations) I've created a new variable y_raw that contains the original values (that construct the error bar) and kept y as the mean values themselves. I'll work out creating a full set of specs for this now.

[sharlagelfand]

Ok, here are example (full) specs with an errorbar at the end for facet and non-facet cases.

I couldn't figure out having errorbars when there are both column and row facets - you can see here it's not quite right! @giorgi-ghviniashvili any idea what might be missing there?

[sharlagelfand]

Just closing the loop on the issue with error bars not appearing with both row/column facets - they are there, just tiny! Related to #54 in terms of zooming in on the final view.

[giorgi-ghviniashvili]

Ok, so I managed to hack error bars for both: faceted and non-faceted views.

https://user-images.githubusercontent.com/6615532/119651664-4af4ca00-be36-11eb-9489-fcc648e472fb.mov

• fixed "1px left" issue
• last layer (from layer array) is animated always which eliminated "point moving up" issue that I showed in the meeting

[sharlagelfand]

Thanks @giorgi-ghviniashvili! I'm running into an error trying to generate with errorbars, in this spec. The error in the console says newSpec.encoding.y.axis is undefined - could you take a look?

[giorgi-ghviniashvili]

@sharlagelfand I will make it work for row + column facets..

[giorgi-ghviniashvili]

@jhofman @sharlagelfand it is not possible to hack a layer that has mark = errorbar with facets, so I am just drawing it with real facets and errorbars..

Only second layer which is not errorbar, and is mark = point, is animated.

But in this case, I see that error bars not correctly aligned..

Unfortunately the fix that I talked yesterday that I hacked all faceted layers only worked for row facets + errorbars, but did not work for column + row facets + errorbars..

[sharlagelfand]

As we talked about in our call, I'll look into manually calculating the error bars and passing via something like rule rather than using mark "errorbar"

[sharlagelfand]

@giorgi-ghviniashvili here is an example spec using rule instead of errorbar

[giorgi-ghviniashvili]

@sharlagelfand @jhofman I fixed alignment issue in hacked faceted view:

[giorgi-ghviniashvili]

This means that if we only hack animated layer and just draw others , will work. Should I still work to fake rule spec?

[jhofman]

looks good!

what was the fix? do you think it will work generally going forward? if so, then no need to try rule i would say.

On Fri, May 28, 2021 at 6:30 AM Giorgi Ghviniashvili < @.***> wrote:

This means that if we only hack animated layer and just draw others , will work. Should I still work to fake rule spec?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/microsoft/datamations/issues/44#issuecomment-850321640, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAATNS7OF4KGACDDTDGC5KTTP5WDVANCNFSM44YYKJOQ .

[giorgi-ghviniashvili]

@jhofman the height was not correctly generating, from -4.7px to 305px, height should have been 309.7px, I was only using 305px for height (assuming that min would be 0..).

I think it will work generally going forward..

[sharlagelfand]

@giorgi-ghviniashvili will this approach work with different sizes? The size of the whole widget can change and each row can be different sizes depending on the number of variables in the row facets

[giorgi-ghviniashvili]

@sharlagelfand , yes I think it will work, we can test bunch of specs if you have.

[sharlagelfand]

@giorgi-ghviniashvili I will remove my logic for rule instead of errorbar, update with the latest code and push the app so we can test scenarios through that

[sharlagelfand]

@giorgi-ghviniashvili I noticed a couple issues with the errorbars for this spec. There is two legends, one for the error bars and one for the points, and the last frame (zoom) does not get animated. When you go to it, it is empty - so maybe not being rendered correctly? Thanks!

https://user-images.githubusercontent.com/15895337/120003559-dd66ac00-bfa3-11eb-9a5b-e235dce34777.mov

[sharlagelfand]

Latest version of the app is here, it seems like the errorbar logic fix works because it is not actually hardcoded as e.g. 305px but rather uses a dynamic value

[giorgi-ghviniashvili]

@sharlagelfand

• Two legends fixed with css:

• about last frame, the domain is NA here:

As soon as you fix "NA" in domain, it will be fixed.

---

Height was not supposed to be hardcoded..

[sharlagelfand]

Thanks @giorgi-ghviniashvili, my bad on NAs in domain! Fixed now, and the legends look good:

https://user-images.githubusercontent.com/15895337/120202068-2319b380-c1f4-11eb-82cf-88bdfd30cf2c.mov

re: height hardcoded, this comment made me think it was - but glad to see it's not!

@jhofman the height was not correctly generating, from -4.7px to 305px, height should have been 309.7px, I was only using 305px for height (assuming that min would be 0..).

I think it will work generally going forward..

[jhofman]

small note on the error bar sequence: right now it looks like the following happens: we're zoomed out, error bars show up, then they disappear as we zoom in, then they reappear.

i realize this is due to the fact that gemini doesn't handle the error bars and we have to fake them, but am wondering if we can do something a bit different so they don't disappear and reappear.

option 1 would be to wait until zoomed to show the error bars, or just fade in the error bars while the zooming in happens?

option 2 would be to somehow fake the animation of them.

option 2 sounds hard. maybe we go with option 1 if that's the case?

[giorgi-ghviniashvili]

@jhofman yes, I know it and this is because we are faking it.

I am just clearing div element where the error bars are mounted, setting opacity to 0 and then fading opacity in (transition). I also have some delay before transition. It fades in just when zooming happens (this means that transition happens concurrently).

Unfortunately I can't do anything different for this, because technically it needs to be deleted and then re-drawn. Otherwise it will not work.

So I think I am doing Option 1 right now.

Also here is the video with delay = 0:

https://user-images.githubusercontent.com/6615532/121074865-7c31aa80-c7e5-11eb-80d9-6f9eb44fb141.mov

Maybe this is a bit better, let me know

---

About option 2, it sounds hard, but also don't get how it should work, we should have another gemini animation under the main animation?

[sharlagelfand]

Looks like there's still an issue with errorbars when there's multiple row facets - I thought we already addressed this but seems like it may have popped up again! For this spec:

[sharlagelfand]

Oh thanks @giorgi-ghviniashvili, just saw this commit! github doesn't seem to notify me of those :) Looks good now.