search

samplchallenges / SAMPL5_logD_PredictionAnalysis

Analysis of all predictions submitted for the distribution coefficient part of the SAMPL5 challenge
MIT License
2 stars 1 forks source link

Decide on/incorporate feedback from Samuel Genheden #11

Closed davidlmobley closed 8 years ago

davidlmobley commented 8 years ago

Genheden feedback

When discussing the different methods used in the submission, citations would help the reader find more reading material. The citation could either be to the original papers or to the papers in the SAMPL5 issue.

I object slightly to the selection of “top performing submissions”. I have no objection for submission 16, but submission 14 and 36 are only really on top if you look at the relative metrics R and tau. I understand that this nitpicking and sort of subjective, but it seems to me that the submissions (apart from 16) on top for RMSE and AUE are not on top for R and tau, and vice versa. Perhaps this should be mentioned in the discussion. As it is written now it is unclear how you found that “two additional submission which performed in the top by these four metrics”.

Discuss as in #10.

bannanc commented 8 years ago

I thought our plan was to cite papers in the SAMPL5 issue once we had a complete list, I think this should be sufficient for his first concern.

So I need to double check my notes, but I'm pretty sure 14 and 36 were the only two that showed up in the top 10 submissions by at least 3 of the 4 metrics considered in this section (R, tau, RMSE, and AUE). This was all in hand written notes on the plots so I'll have to pull them back out and double check.

Why did you reference issue 10?

davidlmobley commented 8 years ago

Why did you reference issue 10?

Sorry, I just meant "the intention in creating this issue is so that we can discuss what changes we ought to make, as I explained in #10 ." :)

I thought our plan was to cite papers in the SAMPL5 issue once we had a complete list, I think this should be sufficient for his first concern.

Yes. Making a note to pull that list for you. Bug me if you don't have it when you need it.

So I need to double check my notes, but I'm pretty sure 14 and 36 were the only two that showed up in the top 10 submissions by at least 3 of the 4 metrics considered in this section (R, tau, RMSE, and AUE). This was all in hand written notes on the plots so I'll have to pull them back out and double check.

I think that's right, but then we should make more clear in the paper why exactly we singled out 14 and 36, i.e. either they are special in some way (they are the only ones that were in the top 10 by at least three of the four metrics) or we just picked them as representative good submissions, and we should make clear which, so that other people who did fairly well don't feel unfairly neglected. :)

bannanc commented 8 years ago

suggestions included in pull request #20