./cabird/interviews.md

[timm]

After review, relabel to 'reviewTwo'. After second review, relabel to 'EditorsComment'.

[wagnerst]

Title of chapter

Interviews

URL to the chapter

https://github.com/ds4se/chapters/blob/master/cabird/interviews.md

Message?

Interviews are a useful research method in software engineering.

Accessible?

The anecdote is a nice start for the chapter. The text is easy to follow and captures in a compact way some nice best practices. I didn't really get why quantitative analysis of interviews is out of the question. It depends on how you ask and how many you ask, I would expect.

Size?

I'm not really sure. There are many, many books, papers, blogs and web sites about interviews. Therefore, parts of the chapter feel a bit lengthy. The Interview Workflow feels superfluous to me. It is rather obvious and not really discussed in the text. I would suggest you go over the chapter reducing it a bit by removing things well covered elsewhere. Maybe you can link or cite good interview method descriptions or guidelines?

Gotta Mantra?

No, there is no mantra. The last section is called Now Go Interview! But that doesn't really fit as a chapter title. Maybe something like To interview is to really understand? I don't have a good idea...

Best Points

I very much like the intro with the anecdote. And the best practices in conducting the interview and post-interview discussion and notes are great.

Minor

• The end of the sentence is missing We also talk about whether we've reached saturation, if and how our interview guide should be modified, and
• Here too: You can find examples of high quality research papers that report on interviews from Emerson Murphy-Hill, Carolyn Seaman, And you should cite such papers.
• _but calculating confidence intervals from interview data probably _is* not a good idea.*

[bramadams]

Title of chapter

Interviews

URL to the chapter

https://github.com/ds4se/chapters/blob/master/cabird/interviews.md

Message?

The chapter motivates and explains the workflow of Chris and his colleagues for planning, executing and analyzing interviews with software engineering professionals.

Accessible?

The chapter is very accessible, and explains in an easy-to-follow way how an SE research group with substantial experience approaches and performs interviews in a software engineering context. The storyline is laid out in a logical way, starting with an anecdote as motivation and leading up to an overview of the different steps necessary to perform interviews in an effective manner. Since only a few steps are SE-specific, many parts of the chapter might also apply outside the context of SE.

The chapter currently only focuses on the benefits of interviews. Hence, something that is currently missing is a short mention of the risks/possible caveats when doing interviews, for example people gaming the questions, low number of people accepting to do an interview, etc.

Size?

I think the major issue with this chapter is its size, since compared to other chapters it is quite long (5 instead of 3 printed pages). This is a pity, as I like the many lessons learnt in the chapter, and almost every workflow step has some hidden nuggets of wisdom.

Some possible parts that could be cut or shortened:

• The discussions of recording and transcription, since in the end the post-interview notes and analysis seem mostly related to the manual notes and impressions.
• The recruitment step and (at least partially) the interview guide step could be shortened and (partially) replaced by a template recruitment email and interview guide put online.
• The image for commit categories could be left out.
• In general, writing could be made less wordy in certain places to save some more space.

I found it difficult, though, to find things to cut, since each step serves its purpose.

Gotta Mantra?

The title just consists of the word "Interviews", which of course covers the essence, but could be a bit more wordy :-)

What about: "Interviewing the Interviewer - How to Interview Software Professionals"? The term "interview" here might be ambiguous (e.g., compared to job interviews), so maybe some kind of qualifier is necessary.

Best Points

Clear overview of workflow for interviews with many nuggets of information.

Weakest Points

Too long compared to other chapters.

Textual Comments

• Page 1: "Many of their code reviews were signed off": I.e., the patch was accepted?
• Page 1: "number of experiments o": I.e., statistical tests, or really experiments?
• Page 1: "hypotheses that we" => "hypotheses for which we"
• Page 4: "if and how our interview guide should be modified, and": The sentence suddenly stopped.
• Page 4: "the post-interview are more coherent" => "the post-interview notes are more coherent"
• Page 5: "from Emerson Murphy-Hill, Carolyn Seaman,": The sentence suddenly stopped.
• Page 5: "probably not a good idea" => "probably is not a good idea"

[cabird]

Thanks for taking the time to read through such a long chapter. I appreciate your advice and help about where and what to cut. That will help me immensely in getting it down to a reasonable size.

Bram, question for you. Do you think the workflow diagram has value or do you agree that it contains obvious stuff?

I was thinking for the title, "Going Straight to the Source: Interviewing in SE Research"

[bramadams]

The text itself is quite easy to follow without diagram, so it could be left out.

That said, I think Stefan is referring to the actual workflow in the text. On the one hand, he's right that many books, blogs, ... have considered the topic of interviews. However, most I've seen are very theoretical and heavyweight in nature (similar to how card sorting relates to official grounded theory), and do not consider the specific context of SE.

So, I like Stefan's suggestion to shorten the commonly known interview steps (maybe referring elsewhere for more info on them) and instead focusing on the unique ones (guide, selection, background, conducting, post and card sort).

The proposed title looks OK.

[prechelt]

2 cents:

• we have few figures overall in the book, so I suggest to leave this one in for its sheer color
• "You can find examples of high quality research papers that report on interviews from Emerson Murphy-Hill, Carolyn Seaman". Maybe I can, but better still would be getting a reference from you.

[lauriew]

@cabird

Glad you've engaged with your reviewers. We're hoping for revised chapters by January 13.

In addition to what the reviewers said, I made the following additional notes: Minor editing:

• ..." I had. All to no avail." I think "all to no avail" is not a sentence so maybe it should be like "... i had -- all to not avail."
• "When we talked, He explained ..." no capital H.
• Say something about the consent form earlier (like as something that leads up to the interview).
• Collecting background data ... explain the x and y axis (in a casual way of course).
• transcription - should you mention transcription software (even if you never used it?)

[CaptainEmerson]

@cabird, some unsolicited feedback...

I always bristle a bit when people talk about sample size. Sample size itself is not a problem, representative-ness is. If you've only interviewed/surveyed/whatever 1 person, but she's totally representative of all developers, great! You're done! No need to sample anyone else.

So, I'd recommend:

Drawbacks of using interviews for research include the usually small sample size

to

Drawbacks of using interviews for research include difficulty in obtaining a broad enough representative sample

And

it is unlikely that you would be able to get a large enough sample

to

it is unlikely that you would be able to get a diverse enough sample

Also, you draw a contrast with random sampling, but you don't name the contrasting method. I think you probably mean "purposive sampling".

[CaptainEmerson]

Also! One thing I wonder about: the "chunked" transcription. I've always wanted to do this, but have been uncomfortable, because it somehow seems less rigorous (and more likely to be rejected by your reviewers). Do you have any references that describe alternatives to full transcription?

[barik]

@CaptainEmerson I think this basically comes down to whether one follows Glaser or Strauss methodologies, both considered to have originally described grounded theory methodology. Glaser did not even record his interviews, and relied on "fields notes." I mostly follow Strauss, because his methodology is more procedural and therefore easier for novices to have confidence that they are "doing it right."

http://web.nmsu.edu/~jalmjeld/EmpiricalResearch/PDFs/transcription_interview_data.pdf