real_data plots for HMM performing better than HSMM

[alexbw]

[alexbw]

[alexbw]

I ran these files in order:

fit_gmm.py
fit_hmm.py
fit_hsmm.py
fit_degraded_models.py
plot_block_predictions.py
plot_onestep_predictions.py

[alexbw]

And then plot_degraded_onestep_predictions.py

[mattjj]

You should update the fitting scripts to use the current parameters and data.

[alexbw]

Will do. But these functioned as we expected before, right?

[mattjj]

The block predictions did (there are images in hipchat somewhere). The onestep didn't, I think.

The scripts probably don't include left-censoring, and they should, otherwise it'll be sensitive to where the test data sequence starts; without left-censoring, the HSMM assumes the data it predicts on to start on a boundary, and if it doesn't, the HSMM will do worse right at the start (which would explain the observations). Still, these scripts were way before all the data improvements, so those should be rolled in.

You can see that all the model code is functioning by running the synthetic scripts. Everything else is just the model performing on data (which is a function of the data and the parameters).

[alexbw]

Yeah synthetic data is working.

[alexbw]

Okay, I'll bring everything else up to speed here.

[mattjj]

I'm looking at how left censoring should work with the predictive_likelihoods and block_predictive_likelihoods functions. I'll get back in a few mins.

[mattjj]

In pyhsmm, I just added kwargs forwarding in predictive_likelihoods and block_predictive_likelihoods so that they can take left_censoring args and they'll just pass it to their states classes, which should make left censoring work:

https://github.com/mattjj/pyhsmm/commit/9235d4f38e227dedc4d63c10025eabe777e86e5d

I'm going to edit the synthetic test scripts in this repo to pass left_censoring=True and make sure it works. When it does, I'll push that version, which will include an updated pyhsmm pointer.

[alexbw]

Thanks. I'll work on packaging up the real data and putting it in the scratch directory. It's all ready, just have to locate it and make sure it matches the semantics of the test files.

[mattjj]

Nice! I'll help with any kinks that happen on the real data tests, too; these scripts haven't been touched in a while.

[mattjj]

Pushed to dev: https://github.com/dattalab/pyhsmm-library-models/commit/b034541d5b023de6c1fac3cc1518b052e584db65

It definitely runs, and it looks like the left censoring is doing what it should:

The bumps in the HSMM plot are gone (compare to this earlier plot) because it doesn't make that start-on-boundary assumption anymore.

[mattjj]

Btw I'm not sure but I think the HMM plot probably dips below the GMM around 15-20 steps into the future because it learned a high self-transition probability to deal with the state memory, but since it doesn't count steps like the HSMM does, it doesn't know when to "turn it off"; that means it has too much of a sticky affinity in a time range when the model should be transitioning.

[mattjj]

We should probably follow up on this again...