Feedback - Proposal : Swap Phase 1's Part 1 and Part 2 for future iterations for PE testing

[9fc70c892]

I don't know if this is the correct place to put feedback and I realize that a large part of this is due to me, student error, not reading instructions clearly, I had assumed that reporting of UG is not allowed in Part 1 thinking "or else what is the point of part 2?".

I understand the rational behind ordering part 1 and 2 this way. It is meant for you to test the program as if you are a first time user (under the scenario of a controlled test, which is done in real world testing) and to test if it works as intended, and not test if you are someone with understanding of what is going on behind it (which is primarily done by developers).

Rationale for this proposal

1. Instructions say: Test based on the Developer Guide (Appendix named Instructions for Manual Testing) and the User Guide

But if there is no Instructions for Manual Testing (or minimal, such as -in my case; all it says is "run using java - jar") in the DG, then student will have to resort to browsing through UG to retrieve all the commands needed to test. This can/will cause them to be focused on UG errors if there are any.

2. So that students have a better idea of what the application is about.

if there are any other individuals who would like to discuss this particular thought, feel free to comment below

[francisyzy]

I don’t think there is a limitation during part 1 to report on functionality bugs only. I spotted some documentation bug but I just noted it down during part 1 instead of reporting it directly due to time constrains in part 1.

But I agree with you, swapping part 1 and 2 will allow the student to spend more time understanding the product via the documentation before trying out the app. However, it is hard for the teaching team to restrict you from testing the jar file in documentation bug phase as you can find their release on Github directly if they don't bundle it with the documents.

[9fc70c892]

I don’t think there is a limitation during part 1 to report on functionality bugs only. I spotted some documentation bug but I just noted it down during part 1 instead of reporting it directly due to time constrains in part 1.

Yeah, thats the student error part on my behalf of not reading the "terms and conditions" clearly. After all, these two information is pretty close to each other. I realized that after submitting.

Though, is there really a need to restrict the testing of jar?

[damithc]

Thanks @9fc70c892 for the suggestion and @francisyzy for inputs.

1. One thing we'll have to improve further is the instructions so that more students don't make the same mistake.
2. The reason for having 1/3 of the testing time dedicated to document bugs is to ensure documents receive sufficient attention during testing, especially the DG. There are several reasons for wanting that, one of which is to ensure everyone is in a position to give an overall evaluation of the project, including quality of documentation.

We'll consider your feedback regarding the order and nature of these two parts for future PE design. Others, feel free to chime in your thoughts as well.

[jefrai]

I agree that there should be

1. a window dedicated to checking documentation, and
2. the window should be at the start of Phase 1

The current system sort of disincentivises reading the UG properly at the start, since it feels more worthwhile to rush for a hands-on feel for the jar before precious Part 1 time runs out. This is a shame, as the UG/DG can reveal quite a lot about how the developers intended for their program to be used, and thus hint at angles of attack. (Anecdotally, I found 1 or 2 additional functionality bugs only while reading the UG carefully in Part 2.)

However, it's not quite feasible to force documentation-only at the start. Banning use of the jar wouldn't be entirely appropriate; some trying out of the program itself is needed to verify/contextualise what the UG says, while reviewing the source code is also needed to check the DG.

Perhaps one way would be to weigh documentation bugs for 1.5x / 2x tester points if submitted within the first 30~45 min? This could prompt students to read the docs first, while still allowing them access to the jar & sources to aid in that.