search

ag-csw / LDStreamHMMLearn

1 stars 0 forks source link

Heatmaps Extended #18

Open alexlafleur opened 7 years ago

alexlafleur commented 7 years ago

We need to understand how timescaledisp and statconc influence the evaluation. So, we would like to have two additional heatmaps for these parameters

Issue #4

alexlafleur commented 7 years ago

performance error

greenTara commented 7 years ago

I don't understand the label on the vertical axis.

On Fri, Oct 28, 2016 at 11:51 AM, Alexandra La Fleur < notifications@github.com> wrote:

[image: performance] https://cloud.githubusercontent.com/assets/4387137/19812750/1c45a2f8-9d37-11e6-9d00-c465e784e9cb.png [image: error] https://cloud.githubusercontent.com/assets/4387137/19812751/1c6d4fe2-9d37-11e6-8928-cef995811c8b.png

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ag-csw/LDStreamHMMLearn/issues/18#issuecomment-256957106, or mute the thread https://github.com/notifications/unsubscribe-auth/ABMMVdt_GqjD2-bfwhCq0qobHE29Q5BAks5q4hnxgaJpZM4Kjbc_ .

alexlafleur commented 7 years ago

I had to shorten the names to avoid an error with the plotting library. (Too long names cannot be plotted) tdisp = timescaledisp conc = statconc

The values 2,4,8 (or 0.125, 1, 8) correspond to the variable values for that heatmap cell.

greenTara commented 7 years ago

Oh, I see - there is some displacement of the labels, but I guess the top row of heatmaps is for eta, the second row is scale_window (could be shortened to swin) and so on.

greenTara commented 7 years ago

We planned to increase the heatmap size anyway. If we do that, there should be more room for the labels.

greenTara commented 7 years ago

Change in strategy on this point - rather than have timescaledisp and statconc on the vertical axis, while taumeta is on the horizontal axis, I would like to have a set of heatmaps for each of these model parameters, with the model parameter on the horizontal axis and the algorithmic parameters (eta, scale_win, numtraj) on the vertical axis.

The objective is to investigate the question of how to optimize the algorithm parameters, given the model parameters. First we try to understand this by varying one model parameter at a time. Later we can look for joint affects of several model parameters, if necessary.

alexlafleur commented 7 years ago

These plots show the heatmaps with timescaledisp on the horizontal axis (8 runs)

performance_tdisp error_tdisp

alexlafleur commented 7 years ago

These plots show the heatmaps with statconc on the horizontal axis (8 runs)

performance_statconc error_statconc

greenTara commented 7 years ago

OK, this looks reasonable for a single run. Once we finalize the taumeta scripts, then we will also run these for a large number of runs to get the expected error.

greenTara commented 7 years ago

Also, we can omit the performance calculation since it does not depend on this parameter in any way.

greenTara commented 7 years ago

The effect from statconc is considerably less than the other parameters (note the error scale goes only from -3 to -2). So it is certainly of lower priority.,

tdisp has a greater effect, but it is somewhat confounded with taumeta, due to the way it was implemented. So this effect would need to be looked at more closely to separate the effect of the spread of timescales from the effect of the magnitude of timescales. So it is put at lower priority because of the effort involved, with likely result that the effect is also negligible.

greenTara commented 7 years ago

The heatmaps that investigate parameters other than taumeta should use the same value of taumeta when generating the data and performing the estimation. Otherwise, more data is generated (by a factor of 2 or more) than is needed, and that makes the computation much slower than necessary.