[Bug]: A MCQ generated becomes multiple question instead of single question

[andhie]

Bug Description

Using the RagDatasetGenerator to generate dataset over a document, i noticed that there a few instance where GPT-4o generate a multiple choice question, but its now treats as individual questions for each choice thus generating improper dataset.

Version

0.11.18

Steps to Reproduce

        dataset_generator = RagDatasetGenerator.from_documents(
            documents=[doc],
            num_questions_per_chunk=20,
            show_progress=True,
        )
        rag_dataset = dataset_generator.generate_dataset_from_nodes()
        rag_dataset.save_json("sample.json")

{
            "query": "**Multiple Choice**: What is the maximum period an employer can suspend an employee from work during an inquiry?",
            "query_by": {
                "model_name": "gpt-4o",
                "type": "ai"
            },
            "reference_contexts": [
                "2. Either party to a contract of service may terminate such contract of service without notice in the event of any wilful breach by the other party of a condition of the contract of service.\n\n**Termination of contract for special reasons**\n\n1. An employer may, on the grounds of misconduct inconsistent with the fulfilment of the express or implied conditions of his service, after due inquiry\u2014\n\n(a) dismiss without notice the employee;\n\n(b) downgrade the employee; or\n\n(c) impose any other lesser punishment as he deems just and fit, and where a punishment of suspension without wages is imposed, it shall not exceed a period of two weeks.\n---\n# Laws of Malaysia\n\n# ACT 265\n\n(2) For the purposes of an inquiry under subsection (1), the employer may suspend the employee from work for a period not exceeding two weeks but shall pay him not less than half his wages for such period:\n\nProvided that if the inquiry does not disclose any misconduct on the part of the employee the employer shall forthwith restore to the employee the full amount of wages so withheld.\n\n(3) An employee may terminate his contract of service with his employer without notice where he or his dependants are immediately threatened by danger to the person by violence or disease such as such employee did not by his contract of service undertake to run.\n\n# When contract is deemed to be broken by employer and employee\n\n15. (1) An employer shall be deemed to have broken his contract of service with the employee if he fails to pay wages in accordance with Part III.\n\n(2) An employee shall be deemed to have broken his contract of service with the employer if he has been continuously absent from work for more than two consecutive working days without prior leave from his employer, unless he has a reasonable excuse for such absence and has informed or attempted to inform his employer of such excuse prior to or at the earliest opportunity during such absence.\n\n# Employees on estates to be provided with minimum number of days\u2019 work in each month\n\n16. (1) Where an employee is employed in any agricultural undertaking on an estate on a contract of service under which he earns wages calculated by reference to the number of days\u2019 work performed in each month of his service, his employer shall be bound either to provide him with work suitable to his capacity on not less than twenty-four days in each month during the whole of which he is so employed, or if the employer is unable or fails to provide work on twenty-four days in each month whereon the employee is willing and fit to work, the employer shall...\n---\n**Employment**\n\nNevertheless, the employer shall be bound to pay the employee in respect of each of such days wages at the same rate as if the employee had performed a day\u2019s work:\n\nProvided that any dispute as to whether an employee was willing or fit to work shall be referred to the Director General for his decision:\n\nProvided further that in computing twenty-four days for the purposes of this subsection, account shall not be taken of more than six days in any week.\n\n(2) A contract of service shall be deemed to be broken by an employer if he fails to provide work or pay wages in accordance with subsection (1).\n\n**Apprenticeship contracts excluded from sections 10 to 16**\n\nSections 10 to 16 shall not apply to apprenticeship contracts which are in a form approved by and of which a copy has been filed with the Director General.\n\n**PART III: PAYMENT OF WAGES**\n\n**Wage period**\n\n(1) A contract of service shall specify a wage period not exceeding one month.\n\n(2) If in any contract of service no wage period is specified, the wage period shall for the purposes of the contract be deemed to be one month.\n---\n**Calculation of Wages for Incomplete Month\u2019s Work**\n\n18A. Notwithstanding section 60I, an employee who is employed on a monthly rate of pay and has not completed a whole month of service shall be paid wages due to him for that month calculated according to the following formula:\n\n- **Monthly wages** \u00d7 **Number of days of the particular wage period** / **Number of days eligible in the wage period.**\n\nThis applies in the following situations:\n(a) where he commenced employment after the first day of the month;\n(b) where his employment was terminated before the end of the month;\n(c) where he took leave of absence without pay for one or more days of the month; or\n(d) where he took leave of absence by reason of having been called up for national service under the National Service Act 1952 [Act 425], to present himself for national service training as required under the National Service Training Act 2003 [Act 628] or to comply with any other written law relating to national service.\n\n**Time of Payment of Wages**\n\n19."
            ],
            "reference_answer": "The maximum period an employer can suspend an employee from work during an inquiry is:\n\n(a) Two weeks",
            "reference_answer_by": {
                "model_name": "gpt-4o",
                "type": "ai"
            }
        },
        {
            "query": "- A) One week",
            "query_by": {
                "model_name": "gpt-4o",
                "type": "ai"
            },
            "reference_contexts": [
                "2. Either party to a contract of service may terminate such contract of service without notice in the event of any wilful breach by the other party of a condition of the contract of service.\n\n**Termination of contract for special reasons**\n\n1. An employer may, on the grounds of misconduct inconsistent with the fulfilment of the express or implied conditions of his service, after due inquiry\u2014\n\n(a) dismiss without notice the employee;\n\n(b) downgrade the employee; or\n\n(c) impose any other lesser punishment as he deems just and fit, and where a punishment of suspension without wages is imposed, it shall not exceed a period of two weeks.\n---\n# Laws of Malaysia\n\n# ACT 265\n\n(2) For the purposes of an inquiry under subsection (1), the employer may suspend the employee from work for a period not exceeding two weeks but shall pay him not less than half his wages for such period:\n\nProvided that if the inquiry does not disclose any misconduct on the part of the employee the employer shall forthwith restore to the employee the full amount of wages so withheld.\n\n(3) An employee may terminate his contract of service with his employer without notice where he or his dependants are immediately threatened by danger to the person by violence or disease such as such employee did not by his contract of service undertake to run.\n\n# When contract is deemed to be broken by employer and employee\n\n15. (1) An employer shall be deemed to have broken his contract of service with the employee if he fails to pay wages in accordance with Part III.\n\n(2) An employee shall be deemed to have broken his contract of service with the employer if he has been continuously absent from work for more than two consecutive working days without prior leave from his employer, unless he has a reasonable excuse for such absence and has informed or attempted to inform his employer of such excuse prior to or at the earliest opportunity during such absence.\n\n# Employees on estates to be provided with minimum number of days\u2019 work in each month\n\n16. (1) Where an employee is employed in any agricultural undertaking on an estate on a contract of service under which he earns wages calculated by reference to the number of days\u2019 work performed in each month of his service, his employer shall be bound either to provide him with work suitable to his capacity on not less than twenty-four days in each month during the whole of which he is so employed, or if the employer is unable or fails to provide work on twenty-four days in each month whereon the employee is willing and fit to work, the employer shall...\n---\n**Employment**\n\nNevertheless, the employer shall be bound to pay the employee in respect of each of such days wages at the same rate as if the employee had performed a day\u2019s work:\n\nProvided that any dispute as to whether an employee was willing or fit to work shall be referred to the Director General for his decision:\n\nProvided further that in computing twenty-four days for the purposes of this subsection, account shall not be taken of more than six days in any week.\n\n(2) A contract of service shall be deemed to be broken by an employer if he fails to provide work or pay wages in accordance with subsection (1).\n\n**Apprenticeship contracts excluded from sections 10 to 16**\n\nSections 10 to 16 shall not apply to apprenticeship contracts which are in a form approved by and of which a copy has been filed with the Director General.\n\n**PART III: PAYMENT OF WAGES**\n\n**Wage period**\n\n(1) A contract of service shall specify a wage period not exceeding one month.\n\n(2) If in any contract of service no wage period is specified, the wage period shall for the purposes of the contract be deemed to be one month.\n---\n**Calculation of Wages for Incomplete Month\u2019s Work**\n\n18A. Notwithstanding section 60I, an employee who is employed on a monthly rate of pay and has not completed a whole month of service shall be paid wages due to him for that month calculated according to the following formula:\n\n- **Monthly wages** \u00d7 **Number of days of the particular wage period** / **Number of days eligible in the wage period.**\n\nThis applies in the following situations:\n(a) where he commenced employment after the first day of the month;\n(b) where his employment was terminated before the end of the month;\n(c) where he took leave of absence without pay for one or more days of the month; or\n(d) where he took leave of absence by reason of having been called up for national service under the National Service Act 1952 [Act 425], to present himself for national service training as required under the National Service Training Act 2003 [Act 628] or to comply with any other written law relating to national service.\n\n**Time of Payment of Wages**\n\n19."
            ],
            "reference_answer": "The context provided does not directly mention \"one week\" in relation to any specific provision or rule. However, it does mention periods of time in various contexts, such as suspension periods and wage calculations. For example:\n\n1. **Suspension Period**: An employer may suspend an employee for a period not exceeding two weeks during an inquiry (Section 2, subsection (2)).\n2. **Wage Calculation**: The calculation of wages for an incomplete month is detailed, but it does not specifically mention a one-week period (Section 18A).\n\nIf you are looking for a specific rule or provision that mentions \"one week,\" the provided context does not contain such information.",
            "reference_answer_by": {
                "model_name": "gpt-4o",
                "type": "ai"
            }
        },
        {
            "query": "- B) Two weeks",
            "query_by": {
                "model_name": "gpt-4o",
                "type": "ai"
            },
            "reference_contexts": [
                "2. Either party to a contract of service may terminate such contract of service without notice in the event of any wilful breach by the other party of a condition of the contract of service.\n\n**Termination of contract for special reasons**\n\n1. An employer may, on the grounds of misconduct inconsistent with the fulfilment of the express or implied conditions of his service, after due inquiry\u2014\n\n(a) dismiss without notice the employee;\n\n(b) downgrade the employee; or\n\n(c) impose any other lesser punishment as he deems just and fit, and where a punishment of suspension without wages is imposed, it shall not exceed a period of two weeks.\n---\n# Laws of Malaysia\n\n# ACT 265\n\n(2) For the purposes of an inquiry under subsection (1), the employer may suspend the employee from work for a period not exceeding two weeks but shall pay him not less than half his wages for such period:\n\nProvided that if the inquiry does not disclose any misconduct on the part of the employee the employer shall forthwith restore to the employee the full amount of wages so withheld.\n\n(3) An employee may terminate his contract of service with his employer without notice where he or his dependants are immediately threatened by danger to the person by violence or disease such as such employee did not by his contract of service undertake to run.\n\n# When contract is deemed to be broken by employer and employee\n\n15. (1) An employer shall be deemed to have broken his contract of service with the employee if he fails to pay wages in accordance with Part III.\n\n(2) An employee shall be deemed to have broken his contract of service with the employer if he has been continuously absent from work for more than two consecutive working days without prior leave from his employer, unless he has a reasonable excuse for such absence and has informed or attempted to inform his employer of such excuse prior to or at the earliest opportunity during such absence.\n\n# Employees on estates to be provided with minimum number of days\u2019 work in each month\n\n16. (1) Where an employee is employed in any agricultural undertaking on an estate on a contract of service under which he earns wages calculated by reference to the number of days\u2019 work performed in each month of his service, his employer shall be bound either to provide him with work suitable to his capacity on not less than twenty-four days in each month during the whole of which he is so employed, or if the employer is unable or fails to provide work on twenty-four days in each month whereon the employee is willing and fit to work, the employer shall...\n---\n**Employment**\n\nNevertheless, the employer shall be bound to pay the employee in respect of each of such days wages at the same rate as if the employee had performed a day\u2019s work:\n\nProvided that any dispute as to whether an employee was willing or fit to work shall be referred to the Director General for his decision:\n\nProvided further that in computing twenty-four days for the purposes of this subsection, account shall not be taken of more than six days in any week.\n\n(2) A contract of service shall be deemed to be broken by an employer if he fails to provide work or pay wages in accordance with subsection (1).\n\n**Apprenticeship contracts excluded from sections 10 to 16**\n\nSections 10 to 16 shall not apply to apprenticeship contracts which are in a form approved by and of which a copy has been filed with the Director General.\n\n**PART III: PAYMENT OF WAGES**\n\n**Wage period**\n\n(1) A contract of service shall specify a wage period not exceeding one month.\n\n(2) If in any contract of service no wage period is specified, the wage period shall for the purposes of the contract be deemed to be one month.\n---\n**Calculation of Wages for Incomplete Month\u2019s Work**\n\n18A. Notwithstanding section 60I, an employee who is employed on a monthly rate of pay and has not completed a whole month of service shall be paid wages due to him for that month calculated according to the following formula:\n\n- **Monthly wages** \u00d7 **Number of days of the particular wage period** / **Number of days eligible in the wage period.**\n\nThis applies in the following situations:\n(a) where he commenced employment after the first day of the month;\n(b) where his employment was terminated before the end of the month;\n(c) where he took leave of absence without pay for one or more days of the month; or\n(d) where he took leave of absence by reason of having been called up for national service under the National Service Act 1952 [Act 425], to present himself for national service training as required under the National Service Training Act 2003 [Act 628] or to comply with any other written law relating to national service.\n\n**Time of Payment of Wages**\n\n19."
            ],
            "reference_answer": "B) Two weeks",
            "reference_answer_by": {
                "model_name": "gpt-4o",
                "type": "ai"
            }
        },
        {
            "query": "- C) Three weeks",
            "query_by": {
                "model_name": "gpt-4o",
                "type": "ai"
            },
            "reference_contexts": [
                "2. Either party to a contract of service may terminate such contract of service without notice in the event of any wilful breach by the other party of a condition of the contract of service.\n\n**Termination of contract for special reasons**\n\n1. An employer may, on the grounds of misconduct inconsistent with the fulfilment of the express or implied conditions of his service, after due inquiry\u2014\n\n(a) dismiss without notice the employee;\n\n(b) downgrade the employee; or\n\n(c) impose any other lesser punishment as he deems just and fit, and where a punishment of suspension without wages is imposed, it shall not exceed a period of two weeks.\n---\n# Laws of Malaysia\n\n# ACT 265\n\n(2) For the purposes of an inquiry under subsection (1), the employer may suspend the employee from work for a period not exceeding two weeks but shall pay him not less than half his wages for such period:\n\nProvided that if the inquiry does not disclose any misconduct on the part of the employee the employer shall forthwith restore to the employee the full amount of wages so withheld.\n\n(3) An employee may terminate his contract of service with his employer without notice where he or his dependants are immediately threatened by danger to the person by violence or disease such as such employee did not by his contract of service undertake to run.\n\n# When contract is deemed to be broken by employer and employee\n\n15. (1) An employer shall be deemed to have broken his contract of service with the employee if he fails to pay wages in accordance with Part III.\n\n(2) An employee shall be deemed to have broken his contract of service with the employer if he has been continuously absent from work for more than two consecutive working days without prior leave from his employer, unless he has a reasonable excuse for such absence and has informed or attempted to inform his employer of such excuse prior to or at the earliest opportunity during such absence.\n\n# Employees on estates to be provided with minimum number of days\u2019 work in each month\n\n16. (1) Where an employee is employed in any agricultural undertaking on an estate on a contract of service under which he earns wages calculated by reference to the number of days\u2019 work performed in each month of his service, his employer shall be bound either to provide him with work suitable to his capacity on not less than twenty-four days in each month during the whole of which he is so employed, or if the employer is unable or fails to provide work on twenty-four days in each month whereon the employee is willing and fit to work, the employer shall...\n---\n**Employment**\n\nNevertheless, the employer shall be bound to pay the employee in respect of each of such days wages at the same rate as if the employee had performed a day\u2019s work:\n\nProvided that any dispute as to whether an employee was willing or fit to work shall be referred to the Director General for his decision:\n\nProvided further that in computing twenty-four days for the purposes of this subsection, account shall not be taken of more than six days in any week.\n\n(2) A contract of service shall be deemed to be broken by an employer if he fails to provide work or pay wages in accordance with subsection (1).\n\n**Apprenticeship contracts excluded from sections 10 to 16**\n\nSections 10 to 16 shall not apply to apprenticeship contracts which are in a form approved by and of which a copy has been filed with the Director General.\n\n**PART III: PAYMENT OF WAGES**\n\n**Wage period**\n\n(1) A contract of service shall specify a wage period not exceeding one month.\n\n(2) If in any contract of service no wage period is specified, the wage period shall for the purposes of the contract be deemed to be one month.\n---\n**Calculation of Wages for Incomplete Month\u2019s Work**\n\n18A. Notwithstanding section 60I, an employee who is employed on a monthly rate of pay and has not completed a whole month of service shall be paid wages due to him for that month calculated according to the following formula:\n\n- **Monthly wages** \u00d7 **Number of days of the particular wage period** / **Number of days eligible in the wage period.**\n\nThis applies in the following situations:\n(a) where he commenced employment after the first day of the month;\n(b) where his employment was terminated before the end of the month;\n(c) where he took leave of absence without pay for one or more days of the month; or\n(d) where he took leave of absence by reason of having been called up for national service under the National Service Act 1952 [Act 425], to present himself for national service training as required under the National Service Training Act 2003 [Act 628] or to comply with any other written law relating to national service.\n\n**Time of Payment of Wages**\n\n19."
            ],
            "reference_answer": "Based on the context provided, the correct answer is:\n\n- C) Three weeks\n\nHowever, the context does not explicitly mention a three-week period. The closest relevant information is about the suspension period, which states that an employer may suspend an employee for a period not exceeding two weeks. Therefore, if the query is about the maximum suspension period, the correct answer should be two weeks, not three weeks.",
            "reference_answer_by": {
                "model_name": "gpt-4o",
                "type": "ai"
            }
        },
        {
            "query": "- D) Four weeks",
            "query_by": {
                "model_name": "gpt-4o",
                "type": "ai"
            },
            "reference_contexts": [
                "2. Either party to a contract of service may terminate such contract of service without notice in the event of any wilful breach by the other party of a condition of the contract of service.\n\n**Termination of contract for special reasons**\n\n1. An employer may, on the grounds of misconduct inconsistent with the fulfilment of the express or implied conditions of his service, after due inquiry\u2014\n\n(a) dismiss without notice the employee;\n\n(b) downgrade the employee; or\n\n(c) impose any other lesser punishment as he deems just and fit, and where a punishment of suspension without wages is imposed, it shall not exceed a period of two weeks.\n---\n# Laws of Malaysia\n\n# ACT 265\n\n(2) For the purposes of an inquiry under subsection (1), the employer may suspend the employee from work for a period not exceeding two weeks but shall pay him not less than half his wages for such period:\n\nProvided that if the inquiry does not disclose any misconduct on the part of the employee the employer shall forthwith restore to the employee the full amount of wages so withheld.\n\n(3) An employee may terminate his contract of service with his employer without notice where he or his dependants are immediately threatened by danger to the person by violence or disease such as such employee did not by his contract of service undertake to run.\n\n# When contract is deemed to be broken by employer and employee\n\n15. (1) An employer shall be deemed to have broken his contract of service with the employee if he fails to pay wages in accordance with Part III.\n\n(2) An employee shall be deemed to have broken his contract of service with the employer if he has been continuously absent from work for more than two consecutive working days without prior leave from his employer, unless he has a reasonable excuse for such absence and has informed or attempted to inform his employer of such excuse prior to or at the earliest opportunity during such absence.\n\n# Employees on estates to be provided with minimum number of days\u2019 work in each month\n\n16. (1) Where an employee is employed in any agricultural undertaking on an estate on a contract of service under which he earns wages calculated by reference to the number of days\u2019 work performed in each month of his service, his employer shall be bound either to provide him with work suitable to his capacity on not less than twenty-four days in each month during the whole of which he is so employed, or if the employer is unable or fails to provide work on twenty-four days in each month whereon the employee is willing and fit to work, the employer shall...\n---\n**Employment**\n\nNevertheless, the employer shall be bound to pay the employee in respect of each of such days wages at the same rate as if the employee had performed a day\u2019s work:\n\nProvided that any dispute as to whether an employee was willing or fit to work shall be referred to the Director General for his decision:\n\nProvided further that in computing twenty-four days for the purposes of this subsection, account shall not be taken of more than six days in any week.\n\n(2) A contract of service shall be deemed to be broken by an employer if he fails to provide work or pay wages in accordance with subsection (1).\n\n**Apprenticeship contracts excluded from sections 10 to 16**\n\nSections 10 to 16 shall not apply to apprenticeship contracts which are in a form approved by and of which a copy has been filed with the Director General.\n\n**PART III: PAYMENT OF WAGES**\n\n**Wage period**\n\n(1) A contract of service shall specify a wage period not exceeding one month.\n\n(2) If in any contract of service no wage period is specified, the wage period shall for the purposes of the contract be deemed to be one month.\n---\n**Calculation of Wages for Incomplete Month\u2019s Work**\n\n18A. Notwithstanding section 60I, an employee who is employed on a monthly rate of pay and has not completed a whole month of service shall be paid wages due to him for that month calculated according to the following formula:\n\n- **Monthly wages** \u00d7 **Number of days of the particular wage period** / **Number of days eligible in the wage period.**\n\nThis applies in the following situations:\n(a) where he commenced employment after the first day of the month;\n(b) where his employment was terminated before the end of the month;\n(c) where he took leave of absence without pay for one or more days of the month; or\n(d) where he took leave of absence by reason of having been called up for national service under the National Service Act 1952 [Act 425], to present himself for national service training as required under the National Service Training Act 2003 [Act 628] or to comply with any other written law relating to national service.\n\n**Time of Payment of Wages**\n\n19."
            ],
            "reference_answer": "The context information provided does not directly mention a period of four weeks. Therefore, based on the given context, there is no relevant information to support the query answer as \"Four weeks.\"",
            "reference_answer_by": {
                "model_name": "gpt-4o",
                "type": "ai"
            }
        }

Relevant Logs/Tracbacks

No response

[dosubot[bot]]

Hi @andhie 👋, I'm here to assist you with any issues, questions, or contributions you have. Let's squash those bugs and find solutions together!

The issue you're experiencing with the RagDatasetGenerator in version 0.11.18, where multiple choice questions are being treated as individual questions, might be related to how the generator processes and formats multiple choice questions. Unfortunately, I couldn't find specific details on how multiple choice questions are handled internally in the RagDatasetGenerator or any known issues related to this in the available documentation ^[1].

To address this, you might consider checking the configuration options and parameters that influence question generation. For instance, you can customize the text_question_template or text_qa_template to better handle multiple choice questions. Additionally, reviewing the logic in your implementation to ensure that the multiple choice questions are correctly formatted before being processed by the generator might help mitigate this issue ^[1][2].

To continue talking to Dosu, mention @dosu.