Paper 1 Referee Report

[drphilmarshall]

Reviewer's Comments:

This work merits publication after some minor revisions. Following the order of the text, my comments (on both form and content) are :

• [x] In the abstract, "expert-classified images which lack lenses" should be replaced by something like "real images that lack lenses" or "observed images that lack lenses", talking about "expert classified images" when referring to non lenses seems a bit confusing.
• [x] Page 9, 4 lines after equation (2), "Nx is the number of lenses the volunteer has been shown" should be replaced by "Nx is the number of training subjects of the relevant type the volunteer has been shown", since it can refer either to lenses or non lenses.
• [x] Page 9, I do not think it is necessary to create a specific subsubsection 4.2.1 to present the examples, which can be presented in the rest of the text of subsection 4.2.
• [x] Page 10, maybe the title of subsection 4.5 : "Practicalities" should be changed for a more specific title here.
• [ ] Page 11, Figure 5, the figure is interesting but it should be better highlighted with clearer color contrasts, for the histograms and for the circles. Issued as #219
• [ ] Page 12, Figure 6, the different linestyles should be replaced by different colors to make the whole plots more readable, and to be able to distinguish the 4 "stars" corresponding to a lens probability threshold of 0.95 in each case. Issued as #220
• [x] Still for this figure, in the caption, "Lens probability varies along the curve" should be replaced by "Lens probability threshold varies..."
• [x] Page 12, in the caption of Figure 7, the authors should add a sentence about the inset as in the caption of Figure 6 : "The inset shows a magnified view of the bottom righthand corner of the plot".
• [x] Page 12, subsection 5.1, paragraph 5, line 9, I would replace "If the training set samples from the same systems as the test set" by "AS the training set samples..."
• [x] Page 15, last paragraph before Section 6, first sentence, there is a repetition of "between".
• [x] Page 16, subsection 6.1, I would write the first sentence of the 3rd paragraph: "The false positives show similar behavior.." at the end of the second paragraph, and begin the 3rd paragraph with "As well as subjects being 'unlucky'...".
• [x] Page 18, the last paragraph of subsection 6.2 quickly deals with the subject of automated lens searches. I would have liked to read more development on that subject, which I could find though in Space Warps II. Nevertheless, I think the authors should at least add a sentence or two about it in their conclusion, maybe by completing the last indent of Section 7 concerning the missed lenses.
• [x] Page 18, I think the title of subsection 6.3 : "Increasing capacity" should be changed to "Increasing crowd capacity" to avoid confusion.
• [x] Page 18, subsection 6.3, 1st paragraph, line 7, there is a 'typo' : "quantitively".
• [ ] The authors should add a discussion about the complexity of computing of a selection function in the case of visual inspection compared to the automated methods.

[aprajita]

On the selection function comment, i read your text - i think it's fine even though it contradicts the referee's implied expectation that the visual selection function is more complex than the automated searches. Couple of minor points when you refer to "labels", I don't think this is very clear as it hasn't been used before... you mean lens, not lens or unclassified, so perhaps classification rather than labels might be more clear?

when you say 'In \PaperTwo we investigate this further by assessing the performance of the \SW system against the (small) set of real lens candidates known to lie in the field' would it be helpful to say we don't compute a selection function there either...in case people are expecting to find that in paper 2

suggest a few minor changes to make it a bit shorter "In future, we could employ a larger training set, to enable the selection efficiency to be characterised as a function of multiple observables. This will be achievable, since the small training set used in the current study was classified around 20 times more than the test set: we could therefore collect many fewer classifications per training image in a much larger training set.

Overall in the PDF online your changes aren't shown in bold face - do you need to do that for the resubmission or is that optional?

[surhudm]

Overall in the PDF online your changes aren't shown in bold face - do you need to do that for the resubmission or is that optional?

Aprajita, we should be able to use scm-latexdiff for creating a diff.pdf between the submitted version and the revised version.

—

Reply to this email directly or view it on GitHub https://github.com/drphilmarshall/SpaceWarps/issues/216#issuecomment-124094479 .

[aprajita]

OK, thanks @surhudm!

[anupreeta27]

@drphilmarshall please check my last question in issue #218 before submission. thanks!