more feedback on the tutorials

[brookehus]

This is all pretty high level! I do not mind at all if you ignore all/most/any of these suggestions. I didn't have any widgets because I just cloned the notebooks from source. FYI, I haven't looked at any updated version of 00. I also apologize if my versions are 24h out of date and therefore some of my suggestions are irrelevant.

01 and in general

• [x] This may be too much of a hassle to do, but for exercises which contain two code blocks, maybe some kind of comment at the top of each box can help differentiate them for any users who don't have the widget. Probably this is redundant with the ones that have #FIXME, but it may be helpful for the solutions to just have a comment at the top saying #Worked solution or something.
• [ ] Some of the long printed lists might be better to do something like,

  for i in my_list:
  print(i)

  instead of just printing it, because then each entry will be its own row and easier to read. For example with describing features, or printing trajectory files. Or use pprint.pprint for this purpose (which is the default, if the last element of a cell is formatted by jupyter - @marscher)

• [ ] It seems like the notebooks would benefit a lot from citations. If someone uses these methods, how will they know what to cite without the notebook linking to the relevant papers? Sorry if the citations are already there and I have missed them.

02

• [x] I read, "This time shift has a default value of 10 steps, but it is always a good idea to specify it explicitly.", but then I see lag=1 specified in the relevant command. Is this correct?
• [x] PCA, TICA, and VAMP are shown, but only the first two are explained.
• [x] When the uniform distance between projection coordinates is produced right before moving onto the ala2 example, I don't have a good understanding of why and when I would want this - there seems to be no explanation provided.
• [ ] In the ala2 plot at the beginning of the example, I think it looks better with logscale=True. Same for the density plot in the exercises with ala2 + PCA as well as ala2 + TICA. I don't feel as strongly about using a logscale for the density in the pentapeptide dataset, but maybe for consistency it's better. Definitely for the lag time plots the logscale is helpful for the density ones.
• [ ] Why do we stop using vamp in the exercises?
• [x] When you run TICA, is it possible to change what the variance cutoff is? How would this be done?
• [ ] With the lag time plots that contain the histograms, the densities, and the free energies, it's difficult for me to figure out what I'm supposed to get out of it; it's a little visually overwhelming.

03

• [x] Something I always think about with pyemma's implementation of the CK test is that it requires a choice of the number of metastable states. Maybe this is worth mentioning or elaborating upon. Is this always implemented using PCCA(+)?
• [x] Some cells take longer than an instant to run. Maybe it's worth a comment sometimes if a cell is going to take especially long? [Could not reproduce here, changed in other NBs]
• [x] It may be worth discussing what to do if the CK test is poor - not necessarily by showing an example of this, but some discussion would be helpful. I feel like I would be unprepared to manage a bad CK test. Should I try a different number of metastable states or turn to my features/decomposition?
• [x] Why choose 4 metastable states in case 2/ the ala2 example? Maybe worth at least a discussion?

04

• [x] When the mean and confidence interval around the first ITS is calculated, does this assume a normal distribution? Maybe it's nice to plot the ITS in case for some reason the Bayesian MSMs produce some sort of bimodal distribution or something.
• [x] I read: "Seeing that four metastable states are a reasonable choice for our MSM, we perform PCCA with this number of states in a follow-up notebook." I can accept this based on the plot but it seems strange without comparison to other options that could be demonstrated to be not a reasonable choice.

05

• [x] I think regular PCCA (Deuflhard, 2000) does not compute distributions: it's discrete/crisp. Then PCCA+ (Deuflhard, 2005) does compute distributions in order to not get artifacts for eigenvectors that are close to zero. I think there is a PCCA++ too but I don't know what that is. However I don't think it makes sense to call what's going on here PCCA without the "+"
• [x] In the last code block, I got:

  /Users/brooke/miniconda3/envs/pyemma/lib/python3.6/site-packages/pyemma/plots/plots2d.py:248: UserWarning: zorder=-1 is not an allowed optional parameter and will be ignored
  ' be ignored'.format(key, kwargs[key]))

• [x] Also, the plot in the last code block looks too narrow to me in the vertical direction

06

• [x] What am I supposed to notice in the timescales_hmsm plot that compares the poor and good discretization? I see that later in the notebook it's explained that HMMs are robust to errors in discretization. But maybe good to foreshadow this so I'm not reading too much into the plots.
• [x] The default approach in pyemma seems to be that HMMs/HMSMs are approximated given a regular MSM. this is not always the case, e.g. Robert's ICML paper that approximates an HMM directly from the raw data. Might be worth at least making this explicit, especially if people are coming into this field from other fields where they've used standard HMMs.
• [x] I get that the hmm is less robust to moderate errors in discretization. but what lesson should I take away from this?
• [x] There was at least one cell that took a loooong time. Maybe worth a user warning, maybe at the beginning of the notebook? Like, > 5-10 minutes for some calculations.
• [x] For hmm_6 I get: [will be fixed in PyEMMA, opened a github issue, cf. below]

  /Users/brooke/miniconda3/envs/pyemma/lib/python3.6/site-packages/pyemma/util/statistics.py:60: UserWarning: confidence interval for constant data is not meaningful
  warnings.warn('confidence interval for constant data is not meaningful')

07

• [x] The title inside of this notebook still says 06
• [x] I read: "Be careful not to use the discrete probabilities as weights: you would then reweight without taking the empiricial distribution of the data into account (right panel)." - this is a little confusing to me. Maybe it helps to make explicit what's used instead? Also typo in 'empirical'.
• [ ] This notebook moved pretty quickly for me! I don't think I could use it to recreate anything...

08

• [x] I read: "The set of microstates which is used for the MSM estimation is stored in the MSM python object and can be retrieved via .active_set. In this example we clearly see that some states are missing.". It would be nice to print out the percent active too!
• [x] This code block throws an error which is expected:

  hmm = pyemma.msm.estimate_hidden_markov_model(cluster_bad.dtrajs, n_states, msm.lag)

  I wonder if a try/except formulation would be better, with a helpful printout, so the user isn't disoriented by the error.

[thempel]

@brookehus Thanks very much!

[cwehmeyer]

In the last code block, I got: /Users/brooke/miniconda3/envs/pyemma/lib/python3.6/site-packages/pyemma/plots/plots2d.py:248: UserWarning: zorder=-1 is not an allowed optional parameter and will be ignored ' be ignored'.format(key, kwargs[key]))

That will be fixed with the next PyEMMA release on conda.

[thempel]

[08] This code block throws an error which is expected:

I'd imagine that it could be more difficult to understand if we put it into a try/except context... If people read the text, it should not be a problem. Closing this.

[06] For hmm_6 I get: UserWarning: confidence interval for constant data is not meaningful warnings.warn('confidence interval for constant data is not meaningful')

Linking this to #37

[cwehmeyer]

Also, the plot in the last code block looks too narrow to me in the vertical direction

This, unfortunately, requires a nontrivial change in PyEMMA.

[thempel]

If I'm not mistaken, we've incorporated or at least discussed why we didn't incorporate the comments here. Citations are covered by #105 . Closing this, feel free to re-open.