Content of Paper

[samm82]

1. As discussed in #73, the main takeaway of this paper should be that testing terminology is inconsistent; should I include any recommendations or hold off for now? I think they would be a better fit in my thesis, especially with the benefit of having more time for data collection and more in-depth analysis and with the limited space I have!
2. Obviously we want to drive home the point that there are significant issues with the current terminology. Do drive this home, should I be including every single discrepancy I come across? I feel like focusing on the "meatier" ones (not just the ones with their own section, but those that are less trivial than "they made an obvious typo") will put the emphasize the significance of these discrepancies, so the takeaway isn't "we need to be more careful when writing out documents" but "we need to understand the differences between test approaches before setting out to communicate that to others".

[JacquesCarette]

1. We should at the very least make all the 'obvious' recommendations, and make explicit where the issue is complex enough that it would be rash to make recommendations
2. You should have table(s) that report on the count(s) of discrepancies, to give an impression of the total. You could classify things by severity in such table(s).

[smiths]

I agree with @JacquesCarette. @samm82 you already have two example recommendations for which you have a rationale and visuals (graphs). You can include those and say that deeper analysis is needed going forward.

You cannot (for space reasons) and should not (for keeping the audiences attention reasons) list all the discrepancies. My guess is that you have around 100 discrepancies. As @JacquesCarette you want to use a table. The fact that you can algorithmically (the graph analysis) find many of these discrepancies is great. Don't get distracted by automation now, but my guess is that you will be able to extend your tool to automatically generate these tables. :smile:

[samm82]

I was hoping on having a bit more done, but as per the advice from @smiths to (hopefully) give @JacquesCarette enough time to give some feedback before Friday (or 8AM on Saturday), I think at least parts of my paper are ready for review. In particular, the "Minor Discrepancies" section will likely have a non-trivial amount of content cut, so don't spend too long reviewing each one! I am planning on making the following changes, so keep that in mind!

• [x] Add a visual/numerical (table? pie chart?) representation of number/types of discrepancies (as mentioned above)
• [x] Filter out more trivial/less confident discrepancies (for length, interest, and impact; also mentioned above):
  • [x] Synonym/parent discrepancies (pp. 5-6)
  • [x] Minor discrepancies (pp. 8-10)
  • [x] Improve formatting:
    • [x] References
    • [x] Add notes in-line
    • [x] Make numbering not ambiguous with IEEE's subsubsection enumeration (see #77)
• [x] Move "test-approach categorization discrepancies" (p. 3) to Discrepancies section?
• [x] Reformat/remove self-parent relations section? (p. 6) (could state implicitly in my performance-related recommendation section)
• [x] Write a conclusion

Some potential areas to cut for more space:

• Functional Testing Discrepancies (pp. 6-7) - lengthy with no clear "recommendation"
• Operational (Acceptance) Testing Discrepancies (p. 7) - a simpler example of what boils down to differing nomenclature
• Out-of-scope Requirements-driven Testing (i.e., the bulleted list on p. 4)
• Static Testing (p. 2) - important to set scope for research, but may not be relevant here

Figured I'd at least give an update along with my next steps; any feedback would be appreciated!

[smiths]

@JacquesCarette is there any chance you can review this on Thursday? I'm not available until Friday and it would be nice if we could give @samm82 some feedback before then.

@samm82 to help @JacquesCarette and me in our review, how many pages are we allowed to have for the paper? I believe there is also a requirement for a double-blind review. Are there any other constraints? I'm assuming you have formatted the paper following the ICSE requirements.

[samm82]

It's not quite in ICSE format, but it will be! There is a requirement for a double-blind review, and I get 10 pages of content + 2 pages of references (currently I'm over, which I'm hoping to tackle with my pruning! 😅) If it doesn't work out to get feedback, no worries! It happens ¯\_(ツ)_/¯

[JacquesCarette]

Yes, I should be able to read this today. There are several sessions that I'm not interested in.

[JacquesCarette]

First bit of (pervasive) advice: write in the present tense, active voice! You use a lot of passive voice and some past tense. Makes the prose heavier.

Second: Use shorter sentences. You over-use long sentences.

I've fixed these things in the abstract for now. I won't fix all these things in the text, but will focus on higher-level things next.

[JacquesCarette]

I was not able to 'make' the paper locally (it assumes Windows because of py, assumes that numpy is installed, and I don't know what else). So I'm editing LaTeX a bit blind.

[JacquesCarette]

I'm leaving some comments as comments in the source code itself. You'll see these show up in the diffs as well.

[JacquesCarette]

We're only allowed 2 pages of references (sigh). So removing all the 'available' links (or at least turning them into named links so that the URL is not displayed but shows as the word 'available' which is a link) will give you quite a bit of space back. We can then see if more needs done.

[JacquesCarette]

There should be a conclusion!

[JacquesCarette]

I've done all that I can do now - go through almost all of it, but couldn't finish the last section. I've pushed all my edits and internal comments.

[JacquesCarette]

@smiths , over to you!

[samm82]

Thanks so much @JacquesCarette! I'm going through your edits now. There's one change in particular I'll pull out to a separate issue for further discussion, but so far everything's looking good! There were only a couple of issues in the LaTeX changes (\namedref{under-terms} instead of \nameref{undef-terms}), so kudos to you! I'll also make a lower priority issue for updating the workflow for compiling this paper/thesis (so that it can be done on multiple systems)

[smiths]

@samm82 I'll review the paper tomorrow (Friday). I should be able to get to it by late morning.

[samm82]

@smiths No rush, but the paper is now at a point where the "only" thing left to do on my list is proofreading! I'll wait on this in case there's any feedback on the content itself (so I don't waste time formatting things that will get removed or references that may change order).

[smiths]

@samm82 I've reviewed the first 3 pages. I'll attach them below. I'll keep going, but you can get started on the revisions on this material if you like. The paper looks good, but there are times when you mention something (like the colour of the different sources) before you say what it means. There are also details of why "we" did things that aren't relevant to a general audience. You can "fake" it like we had this general audience in mind from the start. That is why I removed the material about your work being motivated by our work on generation. It isn't really relevant for the paper and the paper doesn't need this motivation.

I will continue working through the other pages of the paper and give you and update when it is available.

paper.pdf

[smiths]

@samm82 I've finished going through the papers. You won't have time for some of my suggestions. I tried to indicate this when I made the comment. We can worry about the more time consuming comments when we focus on the writing for your thesis itself. Besides the constraint of time, we also have the constraint of space. Some of my suggestions will take too much space. The suggestion to add a Conclusions section isn't an optional suggestion though. I know it will take space, but a paper without a Conclusions section doesn't have much of a chance of acceptance. It will feel to the readers like it is incomplete. Also, some readers skip the discussion and just read the intro and conclusions. I suggested that you could remove one of your recommendations to make some extra space, if needed.

paper.pdf

[samm82]

Round two of revisions mostly done; thank you both for your feedback! There are some finishing touches that I'm going to do before submitting; I'm planning on getting up early tomorrow to polish things off, as well as implement any feedback from you two overnight (more aimed at @JacquesCarette based on the timezone; no pressure on either of you!)

Some notes for @smiths based on his feedback:

• I sometimes use "[]" in quotes, which is done to show an omission of a few letters (e.g., quoting "tested software" as "test[] software"). Would it be better to use ellipses in these cases?
• I didn't include years in the background section where I list "insufficient" taxonomies; IEEE uses the IEEE style (shocker) and while referring to author names seems fine, I haven't seen any cases where the year is referred to as well.
• You mentioned referencing a section when I said "a proposed classification will be given during the analysis stage". I simply removed this sentence, in part since this analysis hasn't happened yet, so there's no section to point to!

My finishing touches:

• [x] Find source for the "snowballing" technique
• [x] Finishing sorting references (in particular Table I)
• [ ] #78
• [ ] Write in present active tense
• [ ] Shorten sentences
• [ ] Use en-dash for page ranges in citations
• [ ] Fix display of "editors" in ISTQB's citation
• [x] CRUCIAL: Suppress author and funding information for double-blindness
• [ ] Unlikely, due to both time and space limitations: add note on the sensitivity factor of using the terms "black-" and "white-box testing" (which I stumbled upon in an IEEE editorial manual)

[samm82]

Note that the file is now named PuttingSoftwareTestingTerminologyToTheTest.pdf (if this is a bad idea, I can revert it 😅).

[smiths]

@samm82 the changes look good. You have done an impressive amount of work in a short time! :smile: I hope ICSE works out, but if it doesn't, we should be able to repurpose this work for another venue.

I had never seen the use of [] for an omission before, but I confirmed that this is the right way to do it. No need to make any changes to your quotation style @samm82. :smile:

Don't worry about not including the year in the natbib style references. The author, or author organization, is fine.

This isn't exactly the same thing, but if with a bit of explanation, you could use this reference to snowball sampling.

For writing in the active tense, the LTeX tool with VSCode has a grammar checker that gives good advice on instances of the passive voice. It boils down to watching for cases where you are using the verb "to be".

Don't stress if you can't get everything on your list done before the deadline tomorrow morning. Submit the best version you can get done with the time you have available.

Good luck!

[smiths]

Note that the file is now named PuttingSoftwareTestingTerminologyToTheTest.pdf (if this is a bad idea, I can revert it 😅).

The title of the file is fine. It will make it easier to find in the future. :smile: The reviewer shouldn't be googling to try to find who wrote the submission.

[JacquesCarette]

Sorry for disappearing - but timezones + flight + jet lag means I've re-appeared online just now. Hopefully this all got sorted out in my 'absence'!

[samm82]

Submitted in time! 🥳 There were a couple of minor tweaks that I would wanted to do with more time, but overall, I'm quite pleased with how this turned out; thank you everyone! I pulled out the remaining work to a separate issue (#80). 😁

[JacquesCarette]

Congrats. Right under the wire!

[samm82]

Congrats. Right under the wire!

Man, you guys like wires 😂 Thanks again! 💖 Closing this issue

[smiths]

Great minds think alike! (and fools seldom differ!) :smile: