does the MLE even exist for our problem? i think so, but are we sure?

[shachen]

I think the problem has a MLE, but the EM does not necessarily find it?

[jovo]

@brian/martin - do you think the MLE for our estimation problem exists? my thought is that, for example, for a mixture model, the MLE doesn't exist, because by setting one of the modes to an exact data point, as the variance goes to zero, the likelihood goes to infinity.

it seems to me that a similar thing could happen in our model, where 1 dimension of x is set to be exactly 1 dimension of y. i don't see why that can't happen?

On Fri, May 29, 2015 at 5:01 PM, shachen notifications@github.com wrote:

I think the problem has a MLE, but the EM does not necessarily find it?

— Reply to this email directly or view it on GitHub https://github.com/shachen/PLDS/issues/13#issuecomment-106935619.

the glass is all full: half water, half air. openconnecto.me, jovo.me, office hours https://www.google.com/calendar/embed?src=e2ktu4lrgul8anp8hclrcminp8%40group.calendar.google.com&ctz=America/New_York

[jovo]

I would guess in a complex model like this, there's at least multiple modes. We know for sure, that without identifiability constraints, that's true. I think the simulations help us identify that the model is at least useful

On Sat, May 30, 2015 at 9:47 PM joshua vogelstein jovo@jhu.edu wrote:

@brian/martin - do you think the MLE for our estimation problem exists? my thought is that, for example, for a mixture model, the MLE doesn't exist, because by setting one of the modes to an exact data point, as the variance goes to zero, the likelihood goes to infinity.

it seems to me that a similar thing could happen in our model, where 1 dimension of x is set to be exactly 1 dimension of y. i don't see why that can't happen?

On Fri, May 29, 2015 at 5:01 PM, shachen notifications@github.com wrote:

I think the problem has a MLE, but the EM does not necessarily find it?

— Reply to this email directly or view it on GitHub https://github.com/shachen/PLDS/issues/13#issuecomment-106935619.

the glass is all full: half water, half air. openconnecto.me, jovo.me, office hours https://www.google.com/calendar/embed?src=e2ktu4lrgul8anp8hclrcminp8%40group.calendar.google.com&ctz=America/New_York

[jovo]

right. but multimodal doesn't necessarily mean MLE doesn't exist, think of a beta distribution with certain parameters.

I think we recover identifiability, but there does not exist an MLE technically.

On Monday, June 1, 2015, Brian Caffo bcaffo@gmail.com wrote:

I would guess in a complex model like this, there's at least multiple modes. We know for sure, that without identifiability constraints, that's true. I think the simulations help us identify that the model is at least useful

On Sat, May 30, 2015 at 9:47 PM joshua vogelstein <jovo@jhu.edu javascript:_e(%7B%7D,'cvml','jovo@jhu.edu');> wrote:

@brian/martin - do you think the MLE for our estimation problem exists? my thought is that, for example, for a mixture model, the MLE doesn't exist, because by setting one of the modes to an exact data point, as the variance goes to zero, the likelihood goes to infinity.

it seems to me that a similar thing could happen in our model, where 1 dimension of x is set to be exactly 1 dimension of y. i don't see why that can't happen?

On Fri, May 29, 2015 at 5:01 PM, shachen <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

I think the problem has a MLE, but the EM does not necessarily find it?

— Reply to this email directly or view it on GitHub https://github.com/shachen/PLDS/issues/13#issuecomment-106935619.

-- the glass is all full: half water, half air. openconnecto.me, jovo.me, office hours https://www.google.com/calendar/embed?src=e2ktu4lrgul8anp8hclrcminp8%40group.calendar.google.com&ctz=America/New_York

the glass is all full: half water, half air. openconnecto.me, jovo.me, office hours https://www.google.com/calendar/embed?src=e2ktu4lrgul8anp8hclrcminp8%40group.calendar.google.com&ctz=America/New_York

[jovo]

Wait, I'm missing the mixture argument now. Wouldn't your mixture argument apply to a sample from a single normal (where we know that the likelihood is finite and the MLE exist)? Is it really true that there are always (boundary cases) of a mixture distribution with infinite likelihood so that no MLE exists? What is the EM algorithm converging to?

The setting I always think about for an unidentified likelihood is a Uniform[theta, theta+1], where the likelihood is exactly flat.

But either way, the model still seems useful.

On Mon, Jun 1, 2015 at 9:23 AM joshua vogelstein jovo@jhu.edu wrote:

right. but multimodal doesn't necessarily mean MLE doesn't exist, think of a beta distribution with certain parameters.

I think we recover identifiability, but there does not exist an MLE technically.

On Monday, June 1, 2015, Brian Caffo bcaffo@gmail.com wrote:

I would guess in a complex model like this, there's at least multiple modes. We know for sure, that without identifiability constraints, that's true. I think the simulations help us identify that the model is at least useful

On Sat, May 30, 2015 at 9:47 PM joshua vogelstein jovo@jhu.edu wrote:

@brian/martin - do you think the MLE for our estimation problem exists?

my thought is that, for example, for a mixture model, the MLE doesn't exist, because by setting one of the modes to an exact data point, as the variance goes to zero, the likelihood goes to infinity.

it seems to me that a similar thing could happen in our model, where 1 dimension of x is set to be exactly 1 dimension of y. i don't see why that can't happen?

On Fri, May 29, 2015 at 5:01 PM, shachen notifications@github.com wrote:

I think the problem has a MLE, but the EM does not necessarily find it?

— Reply to this email directly or view it on GitHub https://github.com/shachen/PLDS/issues/13#issuecomment-106935619.

-- the glass is all full: half water, half air. openconnecto.me, jovo.me, office hours https://www.google.com/calendar/embed?src=e2ktu4lrgul8anp8hclrcminp8%40group.calendar.google.com&ctz=America/New_York

the glass is all full: half water, half air. openconnecto.me, jovo.me, office hours https://www.google.com/calendar/embed?src=e2ktu4lrgul8anp8hclrcminp8%40group.calendar.google.com&ctz=America/New_York

[jovo]

the mixture argument goes as follows:

imagine there are n samples in R, and we are modeling with a mixture of 2 gaussians.

now, initialize with 1 of the n samples being the exact mean of one of the estimated gaussians, and the other n-1 samples comprise the other gaussian.

now, when we estimate the variances, the variance of the gaussian with 1 sample goes to zero, and the likelihood goes to infinity.

of course, this is a pathological example, and i don't see how it can happen with a single gaussian.

but the intuition is that if there is a latent variable whose variance is unbounded, then it is possible for the likelihood to diverge via EM.

in this setting, the goal of EM is not to find the MLE, which does not exist, rather, the goal is to find a good (non-pathological) estimate.

On Mon, Jun 1, 2015 at 9:55 AM, Brian Caffo bcaffo@gmail.com wrote:

Wait, I'm missing the mixture argument now. Wouldn't your mixture argument apply to a sample from a single normal (where we know that the likelihood is finite and the MLE exist)? Is it really true that there are always (boundary cases) of a mixture distribution with infinite likelihood so that no MLE exists? What is the EM algorithm converging to?

The setting I always think about for an unidentified likelihood is a Uniform[theta, theta+1], where the likelihood is exactly flat.

But either way, the model still seems useful.

On Mon, Jun 1, 2015 at 9:23 AM joshua vogelstein jovo@jhu.edu wrote:

right. but multimodal doesn't necessarily mean MLE doesn't exist, think of a beta distribution with certain parameters.

I think we recover identifiability, but there does not exist an MLE technically.

On Monday, June 1, 2015, Brian Caffo bcaffo@gmail.com wrote:

I would guess in a complex model like this, there's at least multiple modes. We know for sure, that without identifiability constraints, that's true. I think the simulations help us identify that the model is at least useful

On Sat, May 30, 2015 at 9:47 PM joshua vogelstein <jovo@jhu.edu>

wrote:

@brian/martin - do you think the MLE for our estimation problem

exists? my thought is that, for example, for a mixture model, the MLE doesn't exist, because by setting one of the modes to an exact data point, as the variance goes to zero, the likelihood goes to infinity.

it seems to me that a similar thing could happen in our model, where 1 dimension of x is set to be exactly 1 dimension of y. i don't see why that can't happen?

On Fri, May 29, 2015 at 5:01 PM, shachen notifications@github.com wrote:

I think the problem has a MLE, but the EM does not necessarily find it?

— Reply to this email directly or view it on GitHub https://github.com/shachen/PLDS/issues/13#issuecomment-106935619.

-- the glass is all full: half water, half air. openconnecto.me, jovo.me, office hours https://www.google.com/calendar/embed?src=e2ktu4lrgul8anp8hclrcminp8%40group.calendar.google.com&ctz=America/New_York

the glass is all full:  half water, half air.

openconnecto.me, jovo.me, office hours https://www.google.com/calendar/embed?src=e2ktu4lrgul8anp8hclrcminp8%40group.calendar.google.com&ctz=America/New_York

the glass is all full: half water, half air. openconnecto.me, jovo.me, office hours https://www.google.com/calendar/embed?src=e2ktu4lrgul8anp8hclrcminp8%40group.calendar.google.com&ctz=America/New_York