Comparison between MBAR and histogram reweighting

[ramess101]

@mrshirts @msoroush @jpotoff

For the JCED FOMMS manuscript, I am comparing the estimated vapor-liquid saturation properties obtained with MBAR and with the standard histogram reweighting (HR) approach. In other words, I am not varying the force field parameters but I am simply calculating rholiq, rhovap, and Psat for the same force field that is used in the GCMC simulations. Here are my preliminary results for branched alkanes:

The exact same configurational snapshots are employed for the MBAR and HR estimates. These configurations (in this case, it is actually just the number of molecules and energies) were obtained in the previous study of Mick et al. for branched alkanes. No new simulations were required to obtain this figure. The HR results are taken from the supporting information of Mick et al. and, therefore, there may be some small differences in how the two approaches distinguish between vapor and liquid phases and/or compute the absolute pressure. But for our purposes, this figure is pretty convincing evidence that MBAR and HR are consistent.

The error bars are those reported by Mick et al. At some point I will need to report uncertainties for MBAR as well. Most likely I will use bootstrap resampling for this.

I know that there are a lot of data points on this figure, and that the different colors, lines, symbols are indiscernible. I am open for suggestions as to how I can better convey the deviations between MBAR and HR. The main point of this figure is just to see the temperature dependence and the range of percent deviations.

[mrshirts]

Can you provide a legend to start to at least try to start discerning things? :)

[ramess101]

@mrshirts

There are 32 compounds, so a legend would be challenging. No real rhyme or reason for the shapes, colors, and lines. I only made them different colors so that it wasn't one massive black blob. Suggestions?

[mrshirts]

Ah, I see, this is combined for 32 molecules.

I like to organize figures around questions I'm answering. So I guess the question is; what questions are you answering in a given figure? That will determine what is plotted.

If the answer is to "show the range of deviations", you could plot a mean and 1 sigma interval of the standard deviation between the methods at each T.

[ramess101]

@mrshirts

Yes, that is really all we need to show. The one complication is that the Tr values are not the same for each compound. But we could easily bin the different Tr ranges and make a box-whisker plot.

[ramess101]

@mrshirts @msoroush @jpotoff

Here are the results for the alkynes:

There does not appear to be any difference in agreement between the branched alkanes (see previous post) and the alkynes.

This is with both branched alkanes and alkynes.

[ramess101]

@mrshirts

I think the box-and-whisker plot is much more helpful. What do you think?

[ramess101]

In the previous plot, the whiskers were set to Q3 + 1.5(Q3-Q1) and Q1 - 1.5(Q3-Q1). All points beyond the whisker are shown as "outliers". Although this is standard convention, I kind of prefer using the whiskers to convey the upper and lower 95% confidence intervals. With this approach, the outlier are usually omitted since there are too many of them (they are really just in the tails, not outliers). This is what a boxplot with whiskers as 95% confidence intervals looks like:

[mrshirts]

Yes, this is much cleaner. Just need to have a clear caption.

[ramess101]

I now have heat of vaporization included in the MBAR-GCMC class. So I will want to repeat this entire analysis for deltaHvap as well.

But we need to resolve the discrepancy in deltaHvap between MBAR and HR first #10

[ramess101]

@mrshirts @jpotoff

I think there are two good ways to visually compare MBAR and HR. The first is to plot the average percent deviation between the two methods, which is what I have shown previously, e.g.:

However, the downside of this approach is that is says nothing about the reported uncertainties in MBAR and HR. That was the only advantage of the original (messy) plot that did include the uncertainties:

The best way to include the uncertainties and have a clean plot (i.e., to not plot all the points but bin them together in a boxplot) is to plot the deviations relative to the uncertainties. Here is an example of what this plot would look like:

Note that currently I still only have the HR uncertainties, but in the future the denominator could be the combined MBAR and HR uncertainty.

The scaled deviation plot tells you if the two methods are consistent within their uncertainties. Specifically, the two methods are consistent when the whiskers (which represent the 95% confidence interval) are less than 1.96. If the whiskers are higher than 1.96 the methods are either inconsistent or the uncertainties were underestimated.

One disadvantage of this type of plot is that the reader loses any sense of how precise MBAR and HR actually are. In other words, the scaled deviation plot can tell us if the two methods are consistent but it doesn't allow the reader to know what the expected deviation would be (i.e., the average percent deviation). Also, the percent deviation plot is a little more intuitive.

Do you prefer the percent deviation or scaled deviation plot? Could we include both? Or maybe just include one in the manuscript and the other in the supporting information?

[mrshirts]

Generally, I like one plot (or subplot, for multiple related points) per point that is made. So nothing wrong with the same data shown in different ways to prove different things.

[ramess101]

@mrshirts @msoroush @jpotoff

I have been playing around with some other ways of representing the agreement between MBAR and HR. This figure shows the absolute value of the relative deviation with respect to the uncertainty (delta_X is the combined uncertainty from MBAR and HR). The solid red line is the analytic expression for the median of a half-normal distribution (since we are taking the absolute value) and the red dashed lines are the analytic expressions for Q1 and Q3. For perfect agreement, the solid red line should match the median and the dashed lines should match the boxes. Clearly the agreement is not perfect, but it is pretty good for most Tr for saturated liquid density, vapor pressure, and heat of vaporization. The largest discrepancy is for saturated vapor density (which impacts Zvap and PDeltaV), where the smaller-than-analytic deviations suggest that the MBAR and/or HR uncertainties in this property are too big.

I don't think I will use this figure because it is less common and requires a long explanation. But I thought it was interesting nonetheless.

[ramess101]

@mrshirts @msoroush @jpotoff

By the way, lowering the tolerance (see Issue #4) did improve the precision of the coexistence estimates. This improved the agreement between MBAR and HR for the simple example where the force field is not changed: