Bias vs. Variance prioritization

[dlee138]

Are there any applications or situations where one might favor lower bias over lower variance, or vice-versa? Ideally, you want both low bias and low variance, but if you can't find a suitable model that can't do this, could you try to raise one while keeping the other one low, instead of making a compromise by raising both by a little?

[ElanHR]

I think most applications make this tradeoff based on the amount of data they have to train on. For example if you have only a couple data points you would want to compensate for this by adding as much domain-specific knowledge you have to your model (eg. we know that our edges are directed and the graph contains no self loops etc.) thus introducing bias in exchange for lower variance. These constraints mean we generalize better but we might not represent the true distribution even if we were given infinite data.

On the flip side if we have infinite - or at least a lot of - data we can better trust that our sample represents the true distribution and we would want our model to be as expressive as possible since we can safely assume that anything we learn from the data isn't just a spurious case due to low sample size. In this case we're decreasing our bias (so we can express whatever patterns we find) in exchange for higher variance (we're dependent on the samples we see).

The bias/variance tradeoff can be sort of be seen as how much we want to trust our data vs our instincts/domain-specific knowledge. Low bias means our model is expressible whereas low variance means our model better generalizes to new unseen data.

PS: Below is a resource I found to be useful with regards to this which had a really good quote: "A gut feeling many people have is that they should minimize bias even at the expense of variance. Their thinking goes that the presence of bias indicates something basically wrong with their model and algorithm. Yes, they acknowledge, variance is also bad but a model with high variance could at least predict well on average, at least it is not fundamentally wrong." http://scott.fortmann-roe.com/docs/BiasVariance.html

[ajulian3]

I read online that when having a high variance, it is optimal to gain more training procedures. However, that would increase the bias. Specifically, the bias is said to be harder to improve than variance. How do we combat this through algorithms?

[kristinmg]

So, is it generally "better" to have a lower bias at the expense of a higher variance?

[jovo]

a couple points here:

1) adding more samples, assuming they are sampled from the assumed distribution, can never add bias, only remove variance 2) there are no general rules for bias vs. variance, it is totally problem dependent. just like the question of whether to buy a lottery ticket is totally context (problem) dependent. your risk functional determines how much relative bias you are willing to trade for variance and vice versa, if not explicitly, than implicitly.

On Fri, Feb 20, 2015 at 10:57 AM, kristinmg notifications@github.com wrote:

So, is it generally "better" to have a lower bias at the expense of a higher variance?

— Reply to this email directly or view it on GitHub https://github.com/Statistical-Connectomics-Sp15/intro/issues/107#issuecomment-75261350 .

the glass is all full: half water, half air. openconnecto.me, we're hiring! https://docs.google.com/document/d/14SApYAzxF0Ddqg2ZCEwjmz3ht2TDhDmxyZI2ZP82_0U/edit?usp=sharing , jovo.me, my calendar https://www.google.com/calendar/embed?src=joshuav%40gmail.com&ctz=America/New_York