Research new data structure for Sample Exam

danopolan commented 4 weeks ago

Recent delivery of CTAL TA Sample Exam for Alpha review raised some questions on using YAML format for all sample questions. It's working fine for simple questions like we mostly have in CTFL module, but for all advance modules, this format is becoming hard to use.

Specifically, the issues are:

Authors do not have Markdown preview while creating questions
Authors make mistakes in indents while working with YAML file
DOCX output is just YAML in DOCX format

I was thinking about splitting questions into separate Markdown files, with the structure like below. Then files can be parsed and merged into a single YAML file within the pipeline.

# metadata
lo: 1.1.1
k-level: K1
points: 1
correct: c

## question 
Which of the following statements describe a valid test objective?

## answers
1. To prove that there are no unfixed defects in the system under test
2. To prove that there will be no failures after the implementation of the system into production
3. To reduce the risk level of the test object and to build confidence in the quality level
4. To verify that there are no untested combinations of inputs

## justification
1. Is not correct. It is impossible to prove that there are no defects anymore in the system under test. See testing principle 1
2. Is not correct. See testing principle 7
3. Is correct. Testing finds defects and failures which reduces the level of risk and at the same time gives more confidence in the quality level of the test object
4. Is not correct. It is impossible to test all combinations of inputs (see testing principle 2)

But this is just a task to discuss the approach.

Witiko commented 4 weeks ago

Would it be OK to make this more case-by-case? For example, if there is a complex explanation for question 6 that an author would like to separate into its own Markdown file, they would write explanation: /question-06-explanation.md (see file transclusion) in file questions.yml and write the explanation into a new Markdown file question-06-explanation.md. This way, simple questions can still be all inside a single YAML file, which increases visibility and simplifies both review and editing, while more complex questions can be offloaded to external Markdown files without increasing the complexity of the CI.

danopolan commented 3 weeks ago

I am collecting feedback on this one from engineers, stakeholders and authors. Just you know that this is being worked on.

danopolan commented 1 week ago

So the preferred solution by various people is to keep questions text on Markdown using specific structure and parsing it as part of CI into YAML.

This approach is key for:

Easy verification of text, tables and images rendering of MD in the Markdown editors
DOCX output which is key for language reviews and translations for some member boards
Less errors in YAML format while written manually
Ability to split questions into separate files by question, chapter or keep them all in one.

So I see several things we need to do here:

prepare the file structure in Markdown for each question
create script to parse MD files into YAML
test and retrofit this to existing questions

This is still for discussion, but the users have spoken :)

danopolan commented 6 days ago

@Witiko can we agree on next steps and format of Markdown for this issue? I would like to start changing current TA Sample Exam question to a new form in parallel, so we can allow language reviewers to start working on it.

Witiko commented 6 days ago

@Witiko can we agree on next steps and format of Markdown for this issue?

Sure. The format in https://github.com/istqborg/istqb_product_base/issues/70#issue-2345726246 seems appropriate for an automatic conversion to YAML.

istqborg / istqb_product_base

Research new data structure for Sample Exam #70