Should we change the overfitting figure?

[Benjamin-Lee]

One thing that I've been thinking about is modifying the overfitting figure, first discussed in #113. Right now, it looks like this:

One thing that is bothering me is that the testing data is different (in a separate range) from the training data. Would it make more sense to have them be in the same range? That way, the overfit model wouldn't be anywhere near the points. We might need to reduce the number of points to ensure that the point (pun intended) of the graph is clear since the plotted points are quite large for readability.

Also, should we remove the xkcd style plotting? I used it at the suggestion of PLOS Comp Bio editor Philip Bourne who wrote:

Using a sketch style conveys to the viewer that the data is approximate, and that it is the higher-level concepts rather than low-level details that are important

and wrote in Ten Simple Rules for Writing a PLOS Ten Simple Rules Article:

There are few scientific articles where you can borrow your plot style from the XKCD comic series (http://xkcd.com/), and this article is one of them.

[pstew]

I concur. Having the figure less busy is a good idea. I like the aesthetic of the figure so I vote to keep it since it visually makes the point that it's an example without having to read a caption.

[Benjamin-Lee]

I modified it to have fewer points and have them in the same range and it seems to be worse:

[cgreene]

I think having some out-of-training-range points is helpful to make the point about generality. The biggest problem with the original figure to me (and this one as well) is that the overfitting is so severe that the orange line doesn't really look like a model at all (from 1-2 it goes off the top and bottom of the chart entirely.

[Benjamin-Lee]

Some experimentation reveals that using an n-4 degree polynomial reduces the off-the-charts-ness for all but the first and last points:

[cgreene]

Personally I prefer this one. What about scattering some testing data within the training interval and also having some outside it?

[Benjamin-Lee]

Something like this?

[cgreene]

Yea - I think this one works for me 😁

[Benjamin-Lee]

Thoughts on no axes ticks or labels?

[cgreene]

Better without arbitrary labels 👍

[Benjamin-Lee]

Ok pushed to #213. @cgreene can you review?

[rasbt]

I think having some out-of-training-range points is helpful to make the point about generality.

I agree with @cgreene . The only concern I have is whether this could be too much at once for the reader. I forgot, do we have page limits? If not, I would just suggest 2 figures, one illustrating overfitting, one illustrating the out-of-training-range points issue.

Thoughts on no axes ticks or labels?

No ticks and values, but how about labels? Like "Input variable" and "Output variable" ?

[agitter]

We could make one figure with a left panel and right panel for overfitting and out-of-training-range issues. That should take up about the same amount of space as the original figure.

[Benjamin-Lee]

Moving the discussion back here from #213. From @rasbt:

You mean this one? It looks kind of old to me, but well, I have probably seen it too many times in the last 6 years :P. Regarding the discussion in the other thread (#210), we may have to figure out whether we want to convey overfitting & out-of-training-range issues in the same figure, or whether we want to have two separate illustrations. Let's resolve this discussion in the other thread first and then revisit this one.

Spontaneously, I would suggest using the other figure as illustration of the out-of-range issue, and this one (modified with XKCD style) can be used as illustration of just overfitting.

I think a two panel figure is a good idea. I don't think we have a page limit, but we may have a figure limit (my other Ten Simple Rules paper didn't have figures, so I never found out). I can get on creating an overfitting-specific figure.

Any thoughts on how the figure should be set up? Like @rasbt's as a classifier? Or like the current one with a linear regression?

[evancofer]

I think the first one is better. Its explicit reference to the train/test split makes it clear why the model is overfitting or not, and that overfitting is an issue regarding generalization from training to testing. In my opinion, the tri-panel does not do this. Moreover, the tri-panel also references the bias/variance tradeoff, which we do not mention anywhere else in the text. Although the bias/variance tradeoff is an important concept, I think we should strive for simplicity at this point, and I am not sure its discussion is absolutely critical to any of the points that we are trying to make.

[Benjamin-Lee]

Sorry @evancofer, when you say the first one, which figure are you referring to? The one as stands presently or the revised version of #213 (pasted in this issue)?

I agree with your point of avoiding discussion of bias and variance if possible (these are “quick tips” after all). If we mention it in the figure we’ll need to explain it in the text so we should keep those terms out of the figure.

[rasbt]

Although the bias/variance tradeoff is an important concept, I think we should strive for simplicity at this point,

I agree. Explaining bias & variance would require getting into decomposing loss functions, which is a bit too technical for the paper. We would just label it as "overfitting and underfitting."

[evancofer]

@Benjamin-Lee The version in master or the version from that PR both look good. I like the changes introduced in #213 as well. However, even without the changes in #213, I think this figure communicates the message more clearly than the tri-panel. That's not to say the tri-panel isn't a good figure. I think the tri-panel could be good in a much longer review, but I don't think we should add more to the text if we don't have to this late in the game.

[Benjamin-Lee]

One advantage of the figure in @rasbt’s book (the tri-panel) is that it uses the exact same data in each panel. Can anyone think of a way to demonstrate overfitting and out of range generalization in two figures with the same data?

I tried doing it in one figure but it got kind of busy (see the PR version above in this thread).

[Benjamin-Lee]

For those following this thread (and not #213), here is an alternative version of the figure in the paper:

It retains the same styling improvements we've discussed here and I've updated the language of the figure caption to reflect that we're talking about out-of-range generalization. @evancofer is right that a tri-panel would work better in a longer review, so I have focused on simplifying and cleaning up the version we already have.