search

hubverse-org / hubEnsemblesManuscript

https://htmlpreview.github.io/?https://github.com/Infectious-Disease-Modeling-Hubs/hubEnsemblesManuscript/blob/master/analysis/paper/hubEnsembles_manuscript.html
Other
1 stars 2 forks source link

Manuscript Comments from Alvaro #71

Closed lshandross closed 1 month ago

lshandross commented 2 months ago

  1. In Section 3.1. it took me a while to figure out why you weren't mentioning output_type_id. I was thinking maybe you could number the three groups of columns (model ID, task IDs and model output representation), so we know you haven't yet reached the "model output representation" section yet. So, in my mind, the third sentence in section 3.1. would read something like this:

    Per hubverse convention, there are three groups of columns, each group serving a specific purpose: (i) the "model ID" denotes which model has produced the prediction, (ii) the "task IDs" provide details about what is being predicted, and (iii) the "model output representation" specifies how the prediction is represented.

    Predictions are assumed to be generated by distinct models, typically developed and run by a modeling team of one or more individuals. (i) Each model should have a unique identifier that is stored in the model_id column. (ii) Then, the details of the outcome can be stored in a series of task ID columns.

    Obviously, this is just one approach. I had thought maybe subheadings, but (i) is very short, which makes subheadings a bit clumsy.

  2. In Table 3, perhaps you should put an asterisk next to sample to clarify that it is not yet supported by hubEnsembles. I know you explain it in the text, but some people like me skim the text and go straight for tables and images.

  3. In Table 4 I would delete the "(columns)" and "(rows)", as I find them more confusing than helpful.

  4. In Table 4 you also reference Section 3.2.2, but there is no such numbered section. I presume you're referencing the "Linear pool" section, so perhaps that should be numbered "3.2.2. Linear Pool."

  5. I don't know if there's much you can do about this, since the results are almost identical for the mean and geometric mean, but in Figure 4, you can hardly see the line for the geometric mean median point estimates. I checked out the results using 2022-11-19 as the reference date, and the difference between the arithmetic and geometric means are bigger, so the lines might be slightly more visible.

  6. On page 17, when I was reproducing the code, it didn't work. I realized it's because the name for the baseline model in forecast_outputs is actually "Flusight-baseline", not "simple_hub-baseline".

  7. In Figure 5, you have an underscore missing in linear_pool.

  8. I didn't run the code in Section 5 because my computer has storage issues, but it looked good!

  9. In Figures 7, 8, 9, I see what you mean with "(columns)" and "(rows)", but I still find it more confusing than helpful, since the top row is different from the other ones. Maybe I'm just not used to seeing that nomenclature. Also, is "colored points" the common term for the colored lines? I only ask because I don't think of them as points, but rather lines with geometric shapes. Again, this just may be a nomenclature I'm not used to. Another question, should you add a small text that says "lower scores are better" for WIS and MAE, and maybe something like "nearness to the black horizontal line is better" (my phrasing needs improvement), or at least "refer to text for interpretation." Again, this goes to my notion that figures should sort of stand alone, even if you didn't read the text.

eahowerton commented 2 months ago

Also, from Melissa: On page 9 the acronym LOP is used without being spelled out at first use, and throughout the rest of the manuscript, both before and after, linear opinion pool is always spelled out.