Modelling Questions for User Evaluation

[dcorsar]

Proposed extensions to the User Evaluation module for questionnaires and questions asked of a user while evaluating a provided explanation

Question class, subclass of Metric for modelling a question presented to the user that requires an input.

Subclasses for Open Question and Closed Question

Open Question will assume response type is to be defined (e.g. text, int, double), and only a single response it to be given (c.f. SingleInputQuestion and MultipleInputQuestion of [1]).

• Point of discussion: Do we need to be more specific (e.g. email, phone number, address) to help use / validation of input? Are there other validation criteria that should be defined - valid response ranges for numerics for example.

A Closed Question will "offer participants an array of potential responses" [2].

There is a modelling choice to be made concerning subclasses of Closed Question.

• model the number of responses that can be provided - e.g. MulipleChoiceClosedQuestion and SingleChoiceClosedQuestion classes as in [1].
• model the type of question: Ranking (with sublcasses for Definite Ranking and Relative Ranking [2]), Likert (rating agreements, single selected answer allowed), Nominal (non-numerical responses, single or multiple selected answers allowed), Rating (numeric scale answers, single selection), etc. and perhaps a more general Selection (e.g. select single/multiple from the following).

Each Question will be part of a Survey describing a list of questions to be asked

• Point of discussion: is the ordering of questions important?

Responses to questions will be captured as defined in the Conceptual Model deliverable. As User_Group is a subclass of user this will allow user groups to be associated with an answer.

Modelling Closed Question Responses (and Answers) It is proposed to define a Response class, individuals of which will define the available responses to be displayed to the participant for Closed Questions. These can be associated with a displayText property defining the text to be shown to the user, and presentationOrderNumber an optional integer defining the position in the list of responses a particular response should be shown in (e.g. 1 (for first), 2 (for second), etc.).

To reduce ambiguity, each Question individual could have a minimumResponseNumber and maximumResponseNumber attribute value, indicating the minimum and maximum number of responses the participant should provide. Most Questions should be associated with this (Ranking questions are the exception). If the question is a single response, the values should both be 1. If the question is optional, the minimumResponseNumber should be 0. If no value is provided, then assume no limit.

[1] The Survey Ontology [2] Gideon, L. (2012). The Art of Question Phrasing. In: Gideon, L. (eds) Handbook of Survey Methodology for the Social Sciences. Springer, New York, NY. https://doi-org.ezproxy.rgu.ac.uk/10.1007/978-1-4614-3876-2_7

[anjanaw]

17

[anjanaw]

• Point of discussion: Do we need to be more specific (e.g. email, phone number, address) to help use / validation of input? Are there other validation criteria that should be defined - valid response ranges for numerics for example.

These categories (email, phone number, address) are not needed for iSee

• Point of discussion: is the ordering of questions important?

Yes.

Each Question will be part of a Questionnaire describing a list of questions to be asked

Is a survey (we have this already) different from a Questionnaire?

Responses to questions will be captured as defined in the Conceptual Model deliverable. As User_Group is a subclass of user this will allow user groups to be associated with an answer.

As I understand, Case description is associated with a user group, an evaluation result is associated with an individual user that belongs to that user group.

[ike01]

I added Question isPartOf Survey in the pull request for @dcorsar's review. I suppose Survey was used to represent a Questionnaire.

[dcorsar]

Survey is good.

Looking at the pull request @ike01 I'm unclear what the role of the Step class is? Can we just use the ResponseOptions class - or maybe a subclass OrderedResponseOptions?

If we add a questionNumber property with range int, that should allow ordering of questions in the survey.

The set of possible responses are currently captured as Key Literal Value Pairs - this could introduce ambiguity. Should we define a subclass Question Response Order Value Pair or similar so that its clear what the role of the key and value properties are, and to differentiate from other key value pairs used elsewhere?

[dcorsar]

I've made those initial changes as outlined above at the https://github.com/isee4xai/iSeeOnto/tree/issue-20 branch - please check.

[anjanaw]

@ike01 can you create a questionnaire instance and send the JSON?

[ike01]

@dcorsar I'm happy with the proposed changes. Didn't use steps eventually and will remove that. Do you mean Order Value Pair as a subclass of Key Literal Value Pair?

@anjanaw I'll get you the json as soon as possible. The representation is not finalised

[dcorsar]

Apols, it doesn't appear to have pushed to github yesterday - yes @ike01 Order Value Pair is subclass of Key Literal Value Pair - have updated the example likert question and added yes/no questions (based on the Hoffman paper examples)

[ike01]

@dcorsar Happy with the changes. I'll merge issue-20 branch to main unless there are other changes