PHPUnit Test Failures in: question_state_gaveup instead of question_state_gradewrong

[rajandangi]

Environment:

• Moodle version: 4.4.1 (Build: 20240610)
• PHP: 8.1.2.1.2.17
• PostgreSQL: 14.12 (Debian 14.12-1.pgdg120+1)
• OS: Linux 6.5.0-1017-aws x86_64
• PHPUnit: 9.6.18

Error Messages:

1) qtype_essayautograde_walkthrough_testcase::test_responsetemplate
Questions is in the wrong state.
question_state_gaveup Object &00000000000076ba0000000000000000 () is not instance of expected class "question_state_gradedwrong".
--- Expected
+++ Actual
@@ @@
-question_state_gradedwrong Object &00000000000076bb0000000000000000 ()
+question_state_gaveup Object &00000000000076ba0000000000000000 ()

/var/www/site/question/engine/tests/helpers.php:911
/var/www/site/question/type/essayautograde/tests/walkthrough_test.php:244
/var/www/site/question/type/essayautograde/tests/walkthrough_test.php:183
/var/www/site/lib/phpunit/classes/advanced_testcase.php:72

2) qtype_essayautograde_walkthrough_testcase::test_responsesample
Questions is in the wrong state.
question_state_gaveup Object &00000000000076ba0000000000000000 () is not instance of expected class "question_state_gradedwrong".
--- Expected
+++ Actual
@@ @@
-question_state_gradedwrong Object &00000000000076bb0000000000000000 ()
+question_state_gaveup Object &00000000000076ba0000000000000000 ()

/var/www/site/question/engine/tests/helpers.php:911
/var/www/site/question/type/essayautograde/tests/walkthrough_test.php:244
/var/www/site/question/type/essayautograde/tests/walkthrough_test.php:187
/var/www/site/lib/phpunit/classes/advanced_testcase.php:72

[rajandangi]

I have found that after the changes you made in this commit commit to the is_similar_text function, these two unit tests are failing. This issue affects not only version 4.4 but also 4.1 (not tested, but likely in all versions).

public function test_responsetemplate() {
    $this->check_editorfield('responsetemplate');
}

public function test_responsesample() {
    $this->check_editorfield('responsesample');
}

To resolve this issue, you can either review the is_similar_text function again or modify the check_editorfield function using one of the following alternatives:

1. Test with an answer containing very random text that will result in less than 10% similarity, for example, "lorem ipsum".

protected function check_editorfield($editorfieldname) {
        global $PAGE;

        // The current text editor depends on the users profile setting - so it needs a valid user.
        $this->setAdminUser();

        // Required to init a text editor.
        $PAGE->set_url('/');

        // Create an essayautograde question.
        $q = test_question_maker::make_question('essayautograde', $editorfieldname);
        $this->start_attempt_at_question($q, 'deferredfeedback', 1);

        $prefix = $this->quba->get_field_prefix($this->slot);
        $fieldname = $prefix . 'answer';
        $response = 'lorem ipsum';

        // Check the initial state.
        $this->check_current_state(question_state::$todo);
        $this->check_current_mark(null);
        $this->render();
        $this->check_current_output(
                $this->get_contains_question_text_expectation($q),
                $this->get_does_not_contain_feedback_expectation());
        $this->check_step_count(1);

        // Save.
        $this->quba->process_all_actions(null, array(
            'slots'                    => $this->slot,
            $fieldname                 => $response,
            $fieldname . 'format'      => FORMAT_HTML,
            $prefix . ':sequencecheck' => '1',
        ));

        // Verify.
        $this->check_current_state(question_state::$complete);
        $this->check_current_mark(null);
        $this->check_step_count(2);
        $this->render();
        $this->check_contains_textarea('answer', $response);
        $this->check_current_output(
                $this->get_contains_question_text_expectation($q),
                $this->get_does_not_contain_feedback_expectation());
        $this->check_step_count(2);

        // Finish the attempt.
        $this->quba->finish_all_questions();

        // Verify.
        $this->check_current_state(question_state::$gaveup);
        $this->check_current_mark(null);
        $this->render();
        $this->assertMatchesRegularExpression('/' . preg_quote(s($response), '/') . '/',
             $this->currentoutput);
        $this->check_current_output(
                $this->get_contains_question_text_expectation($q),
                $this->get_contains_general_feedback_expectation($q));
    }

2. Change the code to the following, using the same answer, as the is_similar_text function is currently returning greater than 10% similarity:

$this->check_current_state(question_state::$gaveup);
$this->check_current_mark(null);

One very strange thing I've noticed is that even very different responses are showing a high similarity percentage. If this is not supposed to happen, it may be worth rechecking the is_similar_text function.