Null hypothesis?

[smiths]

As mentioned when closing issue #12, I have a further question about the p-values. In my initial reading of the paper, I thought the goal to validate the data was to show that the means of the observed and simulated data are statistically consistent. However, the null hypothesis given in the paper, mu_1 - mu_2 = 0, is rejected. This seems to be saying the opposite of what I expected. I might be making a mistake in my understanding, or there is an inconsistency in how the information is presented in the paper.

[luongcn]

Yes, you are right that the null hypothesis is rejected, which proves that there may have difference between mu_1 (observed mean) and mu_2 (simulated mean). In the other word, mu_1 and mu_2 cannot be exactly equal. This issue happens in many climate papers that I and @alsamoua have read, since there are a lot of factors can cause errors in simulated model. In this case, we want to prove that although mu_1 and mu_2 are not equal, there is a way to approximate a range where the difference between mu_1 and mu_2 can fall into with 90 and 95 confident level (by using confident interval approach). Since we have done the hypothesis testing and confident interval for the data, we want to put them all in the paper. I hope this comment will be clear to you.

[smiths]

Hello @luongcn. Thank you for the information. Your explanation makes sense, but the paper doesn't say the same thing as what you just said. The topic sentence for the hypothesis tests paragraph states: "Hypothesis tests were also used to validate the data in twelve stations." Given that the hypothesis was rejected, your test does not validate the data. If this is the case, you should state this in the paper, since it is confusing otherwise.

If I'm reading correctly, in your comment above you suggest that the confidence intervals were used to show that the difference in means isn't likely to be large. This claim is not made in the paper, but it should be. If the confidence interval results mitigate the "negative" finding that the hypothesis was rejected, then it should be presented this way in the paper. The confidence interval paragraph only lists the largest intervals.

[liresearch]

I have to apologize that I didn’t read the hypothesis test part carefully when @luongcn @luongcn added it to the second draft. After reading it carefully, here is what I think:

It is clear that in some tests the null hypothesis was retained while in the others it was rejected. I agree with Dr. Smith that we cannot say “Since the p-values are less than 0.05, the null hypothesis is unlikely to be true”. We should clearly describe how many were rejected and discuss why:

For those that were retained (mainly for the spring, summer and fall months), the differences between observed data and model output are not statistically significant. The results are consistent with the R^2 and RMSE test results. But this doesn’t necessarily validate our model, because for our hypothesis tests, the sample sizes are relatively small (40-50?) and the null hypothesis could be rejected when the sample sizes are bigger. This is why we need to further validate the model using confidence intervals. For those that were rejected (mainly for the winter months), even with relatively small sample sizes, the differences between observed data and model output are statistically significant. Since we couldn’t validate the model through the comparison of the means, we need to further validate the model using confidence intervals.

Please follow Dr. Smith’s suggestions and make this clear in the paper. Also, please provide more details of the hypothesis test, for example, what is the test statistic?

Another thing is, do you think it’s better to provide confident interval tables instead of p-value tables?

[luongcn]

Thank you for all your valuable comments @smiths @liresearch. I will add some detail following Dr. Smith's comments. I will also replace p-value table by confident interval table

[smiths]

Sounds good to me! @luongcn, can you please close this issue once you have pushed the revised version of the paper to the repo.

By the way, you do not need to rename every version of your paper that you push. You can just have one name. The file contents are time stamped by git, so you can always return to a later version of the file. You can easily roll back to a specific date. If you would rather have a named target to roll back to, you can assign a tag to any commit, and then roll back to the tag as needed. If you would rather continue the practice of changing the file names, that is fine, but over time, you will learn to trust the timestamp provided by git. It also easier than coming up with new file names for the same file. 😄

[luongcn]

I have edited and sent the draft to @alsamoua , he will double check my grammar and upload it later. @liresearch , for the test statistic that you mentioned before, we will need more tables if we add them in, because we have done 144 tests and for each test, there are mean and variation. Do you think if we should do this?

[liresearch]

Kevin, I meant the test statistic that you used for the tests. I assume all these 144 tests are t-test and you used the same test statistic, correct?

[luongcn]

Dr. li, I don't think they are the same, because the monthly mean for 50 year period of one station is different from the other station. However, the variances are quite similar but not exactly equal

[liresearch]

We don't have to provide the values of the test statistic, an equation of the statistic would be enough.

[luongcn]

Sounds good! I will add the equation for it.