`navigateNextItem` should wait for feedback before navigating.

[TomWoodward]

if an item generates feedback on completion, that feedback is immediately hidden by the navigation to the next item.

if i set a breakpoint inside navigateNextItem i can see feedback being shown but then it is immediately cleared when the navigation happens.

i'm using https://github.com/1EdTech/qti-examples/blob/master/qtiv3-examples/packaging/feedbackTest/id-fa3ee5852723/Example01-modalFeedback.xml for testing this. (submissionMode must be individual for it to try to do the feedback, which is not what this example's assessment.xml specifies, which is a whole different conversation)

[TomWoodward]

this may be related to another issue where its not possible to trigger response processing without trying to go to a new question. in your monty hall example you add a "continue" button that triggers EndAttempt and handles this, but other examples like this one don't have that button and are impossible to progress in the player. this example can be tested in the player sandbox which has the "end attempt" button

[paulgrudnitski]

Before I get to your point, a small diversion.

As you may have noticed, there are two ways in QTI to End an Attempt.

When submission mode is "individual" then an item controller should invoke the QTI 3 Player's endAttempt method which ends the attempt, collects the item's responses and state, validates the state (if validate responses is set to true), and then runs response processing. Feedback is displayed if showFeedback is set to true.

When submission mode is "simultaneous" then an item controller should invoke the QTI 3 Player's suspendAttempt method which ends the attempt, collects the item's responses and state, validates the state (if validate responses is set to true). However, no response processing is executed.

These two methods (endAttempt and suspendAttempt) are documented here:

https://github.com/amp-up-io/qti3-item-player?tab=readme-ov-file#6-retrieving-item-state

Getting to your point, yes, you are correct about the QTI 3 Item Player Controller's navigation method and how feedback is displayed (it's not) when navigating between items following an endAttempt. This is intentional. The example items which show feedback or are adaptive are not suitable (IMO) in the context of a typical test. Consequently, I've added an End Attempt interaction button to all of the items in this sample set:

https://qti.amp-up.io/testrunner/start/1

On the other hand, if one were to design an item controller such as the Sandbox which does not navigate between items, then yes the player does function properly as you've noted. Such an item controller is typically found in an LTI item player.

On Fri, May 17, 2024 at 10:55 AM Thomas Woodward @.***> wrote:

this may be related to another issue where its not possible to trigger response processing without trying to go to a new question. in your monty hall example you add a "continue" button that triggers EndAttempt and handles this, but other examples like this one https://github.com/1EdTech/qti-examples/blob/master/qtiv3-examples/packaging/feedbackTest/id-9c33d141576e/Example05-feedbackBlock-adaptive.xml don't have that button and are impossible to progress in the player. this example can be tested in the player sandbox which has the "end attempt" button

— Reply to this email directly, view it on GitHub https://github.com/amp-up-io/qti3-item-player-controller/issues/7#issuecomment-2118123460, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG3MLNUGA2GFGO2EEPZBVDZCZABPAVCNFSM6AAAAABH4RPOE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJYGEZDGNBWGA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[TomWoodward]

putting aside the simultaneous submission, i feel like what you're saying is that this project's main navigation, where all submission is done via navigation to another item, is just not compatible with questions that have completion feedback or are adaptive, unless those items have submission controls built into them.

is there a good way to detect if response processing has resulted in showing feedback? would the best way to handle feedback at the player level be to always have a stationary submit button and then show a "next" button when the item's state becomes completed? this would have the downside of requiring two clicks to progress a question even if it doesn't have feedback.

[svarokk]

Hey,

I've had this issue too, and there seems little ways to do that as the "modals" themselves do not store "hidden" attributes, instead they directly add and remove "hide" classes. Maybe I did overlook something, but I came up with a solution (which is hacky in my opinion)

First of i've used Vue's mixin so I don't have to overwrite the Qti3ItemPlayer source code:

Vue.mixin({
    mounted() {
        if ( this.$options.name == 'ModalDialog' ){
            this.isHide = function(){
                return !this.$refs.mymodal.classList.contains( 'show' );
            }

            this.isShow = function(){
                return this.$refs.mymodal.classList.contains( 'show' );
            }
        }

    },
    created() {
        if ( this.$options.name == 'ModalDialog' ){
            const originalHide = this.hide;
            const originalShow = this.show;

            this.isModalVisible = reactive( { value: false } );

            this.hide = function(){
                originalHide.apply( this, arguments );
                this.isModalVisible.value = false;
                this.$emit( 'hide' );
            },

            this.show = function(){
                originalShow.apply( this, arguments );
                this.isModalVisible.value = true;
                this.$emit( 'show' );
            }
        }
    }
});

Then in TestRunner.vue:

I've created 2 new data attributes:

feedbacks: [],
feedbackWatchers: [],

Created a method to list all modals:

getFeedbacks(){
    const assesmentItem = this.qti3Player.item;
    const store = assesmentItem.catalogFactory.store;

    return store?.getFeedbacks() ?? [];
}

Then edit the other navigation methods, i'll give an example with navigateNextItem method (had to get it from source code as my controller is veeeery different now):

navigateNextItem (state) {
    console.log('[NavigateNextItem]', state)

    const navigate = () => {
        this.currentItem += 1
        this.loadItemAtIndex(this.currentItem)
        this.updateButtonState()
    };

    this.handleNavigation( navigate );
}

the handleNavigation method is:

handleNavigation( navigateCallback ){
    let feedbackSpecificCallback = null;

    try{            
        const feedbacks = this.getFeedbacks();

        const feedbacksCollection = [];
        let shownFeedbacksLength = 0;

        if( feedbacks.length ){
            feedbacks.forEach(feedback => {
                const node = feedback?.node;
                const modal = node?.$refs?.modal;

                if( modal.isModalVisible.value == true ){
                    const watcher = this.watchFeedback( modal );

                    feedbacksCollection.push( modal );
                    this.feedbackWatchers.push( watcher );
                    shownFeedbacksLength++;
                }
            });
        }

        if ( shownFeedbacksLength > 0 ) {
            this.feedbacks = feedbacksCollection;

            feedbackSpecificCallback = () => {
                navigateCallback();

                this.$off( 'navigateAfterFeedbacks', feedbackSpecificCallback );
            };

            this.$on( 'navigateAfterFeedbacks', feedbackSpecificCallback );
        }else{
            navigateCallback();
        }

    }catch( error ){
        console.error( '[Handle feedbacks] Error: ' + error + '. [Skipping validation, using navigateCallback method.]' );

        this.feedbackWatchers   = [];
        this.feedbacks          = [];

        if( feedbackSpecificCallback ){
            this.$off('navigateAfterFeedbacks', feedbackSpecificCallback);
        }

        navigateCallback();
    }
}

watchFeedback method:

watchFeedback( feedback ) {
    const checkAllFeedbacksGone = () => {
        const allRemoved = this.feedbacks.every( feedback => feedback.isModalVisible.value == false );

        if ( allRemoved ) {

            console.warn( '[Handle feedbacks] All feedbacks are removed.' );

            console.warn( '[Handle feedbacks] Destroying watchers...' );

            this.feedbackWatchers.forEach( ( unwatch, index ) => {
                console.warn( '[Handle feedbacks] Destroying watcher ' + ( index + 1 ) );
                unwatch();
            } );

            console.warn( '[Handle feedbacks] Emitting navigateAfterFeedbacks...' );

            this.$emit( 'navigateAfterFeedbacks' );

            console.warn( '[Handle feedbacks] Clearing feedback watchers...' );
            this.feedbackWatchers = [];
        }
    };

    return this.$watch(
        () => feedback.isModalVisible.value,
        ( newValue ) => {
            if ( newValue == false ) {
                console.warn( '[Handle feedbacks] Feedback removed.' );
                checkAllFeedbacksGone();
            }
        }, { immediate: true }
    );
}

And I think that's it, let me know if you need for me to explain any of this, I know it's not the best but it definitely works (if I didn't miss anything haha)

[paulgrudnitski]

Yes, this is exactly what I'm trying to convey:

this project's main navigation, where all submission is done via navigation

to another item, is just not compatible with questions that have completion feedback or are adaptive, unless those items have submission controls built into them.

In the context of a typical test with this type of navigation, we can't provide a good UX if all we have are Item navigation buttons and the items sometimes have Modal Feedback (there are 3 types of Feedback in QTI: inline, block, modal).

Consequently, when we choose to include Modal feedback in an item, or when we have an adaptive item, we also have to account for how this will likely hijack the Item navigation experience. If the item is to be presented along with other items then we have to add extra goodies to the item such as an End Attempt Interaction as you've seen in this collection:

https://qti.amp-up.io/testrunner/start/1

This is the approach taken by several educational publishers.

On Mon, May 20, 2024 at 6:48 AM Thomas Woodward @.***> wrote:

putting aside the simultaneous submission, i feel like what you're saying is that this project's main navigation, where all submission is done via navigation to another item, is just not compatible with questions that have completion feedback or are adaptive, unless those items have submission controls built into them.

is there a good way to detect if response processing has resulted in showing feedback? would the best way to handle feedback at the player level be to always have a stationary submit button and then show a "next" button when the item's state becomes completed? this would have the downside of requiring two clicks to progress a question even if it doesn't have feedback.

— Reply to this email directly, view it on GitHub https://github.com/amp-up-io/qti3-item-player-controller/issues/7#issuecomment-2120500938, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG3MLLGOJRCR3I4BCNIAYTZDH5LTAVCNFSM6AAAAABH4RPOE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRQGUYDAOJTHA . You are receiving this because you commented.Message ID: @.***>

[paulgrudnitski]

Just a thought…We could borrow from two existing patterns in the Player.As I was developing the Player component I felt that there would be use cases for a controller to provide its own UX for validation messages as well as Catalog messages.  The Player provides an internal UX for these messages, but a controller can Suppress that Player’s UX and display its own.We could introduce a new “suppress-modal-feedback” property and pass the DOM element(s) of all shown modals in the item attempt payload coming out of the Player at the conclusion of an endAttempt method.  A controller could then do what it wants with such a payload l.On May 20, 2024, at 8:03 PM, svarokk @.***> wrote: Hey, I've had this issue too, and there seems little ways to do that as the "modals" themselves do store "hidden" attributes, instead they directly add and remove "hide" classes. Maybe I did overlook something, but I came up with a solution (which is hacky in my opinion) First of i've used Vue's mixin so I don't have to overwrite the Qti3ItemPlayer source code: Vue.mixin({ mounted() { if ( this.$options.name == 'ModalDialog' ){ this.isHide = function(){ return !this.$refs.mymodal.classList.contains( 'show' ); }

        this.isShow = function(){
            return this.$refs.mymodal.classList.contains( 'show' );
        }
    }

},
created() {
    if ( this.$options.name == 'ModalDialog' ){
        const originalHide = this.hide;
        const originalShow = this.show;

        this.isModalVisible = reactive( { value: false } );

        this.hide = function(){
            originalHide.apply( this, arguments );
            this.isModalVisible.value = false;
            this.$emit( 'hide' );
        },

        this.show = function(){
            originalShow.apply( this, arguments );
            this.isModalVisible.value = true;
            this.$emit( 'show' );
        }
    }
}

});

Then in TestRunner.vue: I've created 2 new data attributes: feedbacks: [], feedbackWatchers: [],

Created a method to list all modals: `getFeedbacks(){ const assesmentItem = this.qti3Player.item; const store = assesmentItem.catalogFactory.store; return store?.getFeedbacks() ?? [];

},` Then edit the other navigation methods, i'll give an example with navigateNextItem method (had to get it from source code as my controller is veeeery different now): navigateNextItem (state) { console.log('[NavigateNextItem]', state)

const navigate = () => {
    this.currentItem += 1
    this.loadItemAtIndex(this.currentItem)
    this.updateButtonState()
};

this.handleNavigation( navigate );

},

the handleNavigation method is: `handleNavigation( navigateCallback ){ let feedbackSpecificCallback = null; try{
const feedbacks = this.getFeedbacks();

const feedbacksCollection = [];
let shownFeedbacksLength = 0;

if( feedbacks.length ){
    feedbacks.forEach(feedback => {
        const node = feedback?.node;
        const modal = node?.$refs?.modal;

        if( modal.isModalVisible.value == true ){
            const watcher = this.watchFeedback( modal );

            feedbacksCollection.push( modal );
            this.feedbackWatchers.push( watcher );
            shownFeedbacksLength++;
        }
    });
}

if ( shownFeedbacksLength > 0 ) {
    this.feedbacks = feedbacksCollection;

    feedbackSpecificCallback = () => {
        navigateCallback();

        this.$off( 'navigateAfterFeedbacks', feedbackSpecificCallback );
    };

    this.$on( 'navigateAfterFeedbacks', feedbackSpecificCallback );
}else{
    navigateCallback();
}

}catch( error ){ console.error( '[Handle feedbacks] Error: ' + error + '. [Skipping validation, using navigateCallback method.]' );

this.feedbackWatchers   = [];
this.feedbacks          = [];

if( feedbackSpecificCallback ){
    this.$off('navigateAfterFeedbacks', feedbackSpecificCallback);
}

navigateCallback();

}

} watchFeedback method: watchFeedback( feedback ) { const checkAllFeedbacksGone = () => { const allRemoved = this.feedbacks.every( feedback => feedback.isModalVisible.value == false ); if ( allRemoved ) {

    console.warn( '[Handle feedbacks] All feedbacks are removed.' );

    console.warn( '[Handle feedbacks] Destroying watchers...' );

    this.feedbackWatchers.forEach( ( unwatch, index ) => {
        console.warn( '[Handle feedbacks] Destroying watcher ' + ( index + 1 ) );
        unwatch();
    } );

    console.warn( '[Handle feedbacks] Emitting navigateAfterFeedbacks...' );

    this.$emit( 'navigateAfterFeedbacks' );

    console.warn( '[Handle feedbacks] Clearing feedback watchers...' );
    this.feedbackWatchers = [];
}

};

return this.$watch( () => feedback.isModalVisible.value, ( newValue ) => { if ( newValue == false ) { console.warn( '[Handle feedbacks] Feedback removed.' ); checkAllFeedbacksGone(); } }, { immediate: true } );

},` And I think that's it, let me know if you need for me to explain any of this, I know it's not the best but it definitely works (if I didn't miss anything haha)

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[TomWoodward]

i'm considering the requirement of hiding answer information from the client, and if i were going to try to move response processing and item templating to the server, at that point it would probably be trivial to detect if feedback were being added and require pressing next again to continue to the next question.

i'm just trying to figure out how much i care because i'm not sure the player really supports async templating and the use case for feedback/adaptive in a summative assessment where you care about protecting answers may not be that big

[TomWoodward]

@paulgrudnitski let me try to triage the various thoughts i've been raising in this issue. the core of it, i think, the part that could be considered a bug, is that it attempts to show feedback and it just doesn't work.

i think it might be appropriate to either...

if keeping the player navigation as-is, then non-adaptive feedback is unsupported, and could be stripped or suppressed from non-adaptive items

or...

it could be argued that the "next" button on individual submissions is a confusing ui, because students may sometimes be allowed to switch questions without submitting, and a next button might cause some students to submit without intending to. it might be more clear just to have a separate submit button in the ui, which solves several problems

the player also, relatedly, does not support adaptive questions that have no embedded qti-end-attempt-interaction, which are a real thing. having a separate submit button addresses this. you could even change the label of it to "continue" or something if the item is adaptive.

i'm not sure if this is really the best plan, but you could potentially accomplish this by injecting your endattempt-controller-bar element into the item if the submission mode is individual and it doesn't already contain a qti-end-attempt-interaction. this creates a consistent UI experience for submitting while respecting the item format if defined.

i understand that the player-controller isn't necessarily supposed to be a drop-in testing ui, more of an example implementation, but it seems valuable to try to get this right as part of the example.

[paulgrudnitski]

Being mindful that the QTI 3 Item Player Controller - and also the Sandbox

• is intended as a demonstration of capabilities of the QTI 3 Item Player component, I believe that the Controller is very successful in this regard.

If anyone wants to use the Controller as a basis for other use-cases then I think that's wonderful and completely encouraged.

For example, it might make the most sense in your UX to introduce a separate "Submit" button that separates item navigation from ending or suspending an attempt. This is essentially what the "End Attempt" button does on the Sandbox where there are no navigation buttons.

In regard to the comment about Adaptive items, the Player component exposes the item's isAdaptive attribute. Consequently, a controller that has a "Submit" button can examine an item's isAdaptive attribute and then continue to evaluate the value of the built-in QTI outcome variable "completionStatus". Submit should stay enabled (acting like a "continue" button) until completionStatus has the value of "completed" - as per the QTI standard.

Here, in the famous "Monty Hall - Take 2" item, we can see that "completionStatus" is not yet set to "completed" after hitting Submit the first time.

[image: image.png]

But after further interaction and hitting Submit a second time, we can see that the response processing has set the completionStatus variable to "completed", which means that there are no further adaptations.

[image: image.png]

At that point, modal feedback is displayed, and the Submit button should probably be disabled as it's pointless to continue to end the attempt.

I could be wrong, but I'm confident the Player has all of the capabilities for adaptive items that one would expect.

On Wed, May 22, 2024 at 6:59 AM Thomas Woodward @.***> wrote:

@paulgrudnitski https://github.com/paulgrudnitski let me try to triage the various thoughts i've been raising in this issue. the core of it, i think, the part that could be considered a bug, is that it attempts to show feedback and it just doesn't work.

i think it might be appropriate to either...

if keeping the player navigation as-is, then non-adaptive feedback is unsupported, and could be stripped or suppressed from non-adaptive items

or...

it could be argued that the "next" button on individual submissions is a confusing ui, because students may sometimes be allowed to switch questions without submitting, and a next button might cause some students to submit without intending to. it might be more clear just to have a separate submit button in the ui, which solves several problems

the player also, relatedly, does not support adaptive questions that have no embedded qti-end-attempt-interaction, which are a real thing. having a separate submit button addresses this. you could even change the label of it to "continue" or something if the item is adaptive.

i'm not sure if this is really the best plan, but you could potentially accomplish this by injecting your endattempt-controller-bar element into the item if the submission mode is individual and it doesn't already contain a qti-end-attempt-interaction. this creates a consistent UI experience for submitting while respecting the item format if defined.

i understand that the player-controller isn't necessarily supposed to be a drop-in testing ui, more of an example implementation, but it seems valuable to try to get this right as part of the example.

— Reply to this email directly, view it on GitHub https://github.com/amp-up-io/qti3-item-player-controller/issues/7#issuecomment-2124873685, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG3MLLJY2SI2NPMLSRRYY3ZDSQFZAVCNFSM6AAAAABH4RPOE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRUHA3TGNRYGU . You are receiving this because you were mentioned.Message ID: @.***>

[TomWoodward]

i don't explicitly disagree with anything you said, but in the player controller project i think you'll find that it does not support adaptive items that don't contain a built in qti-end-attempt-interaction

i ran into these in the qti-examples repo and i can track them down if you like, but i think it would be best if the player controller's example test flow handled these items, do you not agree?

[paulgrudnitski]

I agree with your statement that the current Player Controller "collection" navigation buttons do not properly support adaptive items. Those buttons also do not properly support new template generation.

The current functionality of the navigation buttons is completely intentional and influenced by the real-world use-case of at least one very prominent educational publisher with over 100,000 adaptive/templated items. For adaptive/templated items, that educational publisher is focused on formative assessment settings where the collections of items are used to support instruction. Each of the 100,000 items has its own end attempt interaction so as to keep the adaptive capabilities separate from collection navigation. Clearly, I have agreed with this educational publisher that adding Submit and New Template buttons to the navigation button UX is not the best UX.

On the other hand, we probably agree that it would be good if we could add all of the QTI examples - which were developed almost 20 years ago - somewhere into the app. For this, I would suggest perhaps the Sandbox is a more suitable place because there are no navigation buttons. There are End Attempt and New Template buttons, as well as additional controls over item session settings.

[image: image.png]

Isn't the Sandbox a better place to demonstrate the Player's capabilities for the QTI Sample items that are templated, or adaptive and templated?

On Fri, May 24, 2024 at 1:25 PM Thomas Woodward @.***> wrote:

i don't explicitly disagree with anything you said, but in the player controller project i think you'll find that it does not support adaptive items that don't contain a built in qti-end-attempt-interaction

i ran into these in the qti-examples repo and i can track them down if you like, but i think it would be best if the player controller's example test flow handled these items, do you not agree?

— Reply to this email directly, view it on GitHub https://github.com/amp-up-io/qti3-item-player-controller/issues/7#issuecomment-2130307379, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG3MLL47GI4XQRNEXU7VZLZD6O25AVCNFSM6AAAAABH4RPOE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZQGMYDOMZXHE . You are receiving this because you were mentioned.Message ID: @.***>

[TomWoodward]

i think the sandbox is a great resource, but i still think that the usefulness of the test runner as an example of test ui is limited pretty severely if you make assumptions about the structure of the qti items.

i don't see completion feedback or minimally formatted adaptive items as particularly fringe use cases.

it seems like you don't really want to update it, which is fine, this is just my opinion, and maybe your new test runner project will address my concerns. feel free to close the issue if you don't intend to make any changes.

[paulgrudnitski]

I built the Player Controller - which first started as the Test Runner only, but has now evolved into the Test Runner and the Sandbox - as an app that developers and QTI authors could use to examine and exercise the methods, properties, and capabilities of the QTI 3 Item Player component and of QTI 3 itself. For these two purposes, the Player Controller has been a massive success (according to the overwhelming feedback I've received).

But I also agree with you that there are scenarios - fringe or not - where the Test Runner part of the Player Controller app may not be a suitable UX.

As you know, the Player and the Player Controller are completely free, open source, and MIT-licensed. I would encourage you to use these repo's and adapt them to your use-case(s).

On Tue, May 28, 2024 at 7:00 AM Thomas Woodward @.***> wrote:

i think the sandbox is a great resource, but i still think that the usefulness of the test runner as an example of test ui is limited pretty severely if you make assumptions about the structure of the qti items.

i don't see completion feedback or minimally formatted adaptive items as particularly fringe use cases.

it seems like you don't really want to update it, which is fine, this is just my opinion, and maybe your new test runner project will address my concerns. feel free to close the issue if you don't intend to make any changes.

— Reply to this email directly, view it on GitHub https://github.com/amp-up-io/qti3-item-player-controller/issues/7#issuecomment-2135291203, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG3MLO3KDRZ3IP4NKGEFS3ZESEWXAVCNFSM6AAAAABH4RPOE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZVGI4TCMRQGM . You are receiving this because you were mentioned.Message ID: @.***>