consider using same level of transparency in all sub-panels of Figure 3.1

[elray1]

Currently, panel (c) has "fully opaque" lines for the component model cdfs, while panels (a) and (b) have some transparency.

[lshandross]

@eahowerton and I made the conscious choice to have the cdfs for panels a and b be partially transparent instead of fully opaque like in panel c to represent the fact that we're initially working with quantiles, NOT a full cdf distribution, and need to estimate a full cdf distribution to get the resulting quantiles from a linear pool ensemble. If you or others have wording suggestions for the figure caption to make this more clear, we'd be happy to implement them.

[eahowerton]

@lshandross I updated the figure according to @elray1's other suggestion (#17), and I think it's clearer if we have the same transparency across all panels. I've updated the figure accordingly. Let me know what you think.

[lshandross]

I kind of like the original figure more, but I'm ok with deferring to others' opinions if majority says that the new version looks better and is clearer in terms of what we're trying to say

[eahowerton]

@lshandross could you elaborate on why you prefer the original figure? Maybe it'd be helpful to list the pros/cons of each. Perhaps then we could find a happy medium?

I tend to find the consistency/simplicity of the new version to be helpful in getting the message across. I think the primary thing we want the readers to take away is that the quantile average has "horizontal" averaging, and the probability average has "vertical" averaging (as shown by the dashed lines). In my opinion, the interpolation point is more secondary, and still made by the difference between circles/squares.

But please disagree with me! After all, we're getting into pretty subjective territory :)

[lshandross]

I guess I was weighing the interpolation as holding a similar level of importance to the "direction" of the averaging for a quantile vs probability average, which is why I liked the different transparencies. I also find the circles and squares to be kind of confusing (since they don't look that clearly different), especially without a legend explaining them. But the second point could be solved by using another shape, like hollow circles, for the probability values to be averaged, which also probably helps differentiate that these are interpolated rather than being the original quantiles

[eahowerton]

That's a good point. I did debate what shape to use, because I was having similar thoughts that the squares weren't very discernible. Let me try your suggestion (including adding a legend). Thanks!

[eahowerton]

@lshandross let me know what you think of the new version. I really like the open circles and the addition of the legend. I tend to think maybe the points on the ensemble distribution should be filled though, do you agree? any other thoughts?

[lshandross]

I see what you mean about the points on the ensemble distribution seeming like they should be filled. What about making them a different filled shape for all three distributions? I think not having them be the same shape as the original quantiles for the last panel is important to avoid confusion

[eahowerton]

I agree that the same points is potentially confusing. I also worry that having a third shape could be as distracting as helpful. What do you think about no points on the ensemble? (I don't love this option either, because we want to signify what is returned, but seems there are cons with all possibilities!)

[lshandross]

I don't like the no points option either. I think that probably we should either leave it as is or try out the third shape option (and/or get additional opinions)

[eahowerton]

Sorry one more idea here - what about having two sets of points (open and close) on the ensemble in the LOP just like the individual models? We could rename the legend to something other than "submitted" for the solid points. Maybe this would be the most accurate representation as well (since we interpolate back to quantiles after calculating the LOP)?

[lshandross]

Hmm that could work, though I do worry the plot could end up looking a bit busy. We could also try removing the solid quantile dots in the third panel.

Maybe it would be worth summarizing all of our different options with pros and cons for each, then asking for feedback since we have a lot of choices without a clear winner.

[eahowerton]

Alright, here are a few options. Let me know if I'm missing one.

fig1-opt1-all-LOP-circles.pdf fig1-opt2-no-solid-LOP-circles.pdf fig1-opt3-no-solid-circles.pdf fig1-opt4-no-ensemble-circles.pdf fig1-opt5-no-ensemble-circles-no-solid-LOP-circles.pdf

Feel free to vote for your favorite, I think my vote is option 4. Also perhaps if needed, we could take this discussion offline? Just wondering if that could be more efficient for this decision.

[lshandross]

I think I'm inclined to agree with you about option 4, now looking at all of them. Not that we need another option, but the one other idea I have would be to remove the open circles entirely from panel 3. Also happy to talk synchronously if we feel like the issue needs more discussion.

[eahowerton]

If we removed the open circles, would we keep the vertical lines? I find these vertical/horizontal lines to be super helpful in understanding the difference between the two methods. I'm not sure it'd be helpful to have the lines without the points, but always happy to try.

Seems like, maybe though, we're on the same page about option 4? I'm happy to implement this if it sounds good to you.

[lshandross]

Yes, we would keep the lines. I was thinking it might be able to imply the interpolation since the cdf lines are semi-transparent. Perhaps we could look at the two option side-by-side, then make a final choice

[eahowerton]

Hopefully I understood your suggestion about the open circles correctly (let me know if you had something else in mind!) Here are the two options I think we're comparing. If we choose option 6, I'd remove the legend.

fig1-opt4-no-ensemble-circles.pdf fig1-opt6-no-ensemble-circles-no-open_LOP-circles.pdf

I also like option 6, but maybe still a preference for option 4. I'm happy either way if you have a stronger opinion.

[lshandross]

Yes, option 6 is what I was imagining. Thanks for drafting all of these.

I think I'm inclined to agree with you about option 4 being a little bit better, so let's go with that. We'll also probably be getting a final look from others once all the changes are done, so we can leave it as option 4 and see what they say.

[eahowerton]

Okay great, sounds like a good plan. I'll make the updates. Thanks for all the thoughts on this @lshandross!