Epic 7.7.1: Syllabus Generator - Logic Legion

[chitoroagad]

Syllabus Generator

Summary:

• This tool aims to create a sample syllabus customised to the liking of the user.
• Users can specify a small number of specific inputs as well as vague instructions to the tool and receive a tailored syllabus.
• Users can specify which sections of the syllabus to include or exclude in their output using the options api parameter.
• The generated syllabus includes the following:
  • Title
  • Overview
  • Policies and Exceptions
  • Grading Policy
  • Assessment Components
  • Required/Recommend Materials

Changes:

1. syllabus_generator directory in features/ containing:
   • tools.py includes SyllabusBuilder class responsible for all logic involved in using the LLM to create the syllabus.
   • model.py includes the Pydantic model for validating output.
   • core.py includes the executor function.
   • metadata.json includes expected inputs for the api.
   • prompt/ includes various text files for prompt generation.
   • tests/ includes test_sb.py used for testing all components of the SyllabusBuilder as well as test.json which includes a sample input for the api.
2. Updated the tools_config.json to be able to access the syllabus_generator straight from the api.
3. Created development environment in flake.nix for better LSP support and simplified environment set up.
   • Activate using nix develop and --command <prefered shell> to enter env.
   • bash is default.

Example:

Input

    {
        "user": {
            "id": "string",
            "fullName": "string",
            "email": "string"
        },
        "type": "tool",
        "tool_data": {
            "tool_id": 2,
            "inputs": [
                {
                    "name": "subject",
                    "value": "Data Structures"
                },
                {
                    "name": "grade_level",
                    "value": "University"
                },
                {
                    "name": "course_overview",
                    "value": "This course covers the fundamental concepts and applications of data structures in computer science. Students will explore various data structures such as arrays, linked lists, stacks, queues, trees, and graphs. The course will emphasize both the theoretical and practical aspects of data structures, including their implementation and analysis of their efficiency. By the end of the course, students will be equipped with the knowledge to use data structures effectively in real-world applications and advanced computing challenges."
                },
                {
                    "name": "customisation",
                    "value": "I want the course to require proficiency in Python"
                },
                {
                    "name": "options",
                    "value": [
                        "all"
                    ]
                }
            ]
        }
    }

Output

{
  "data": {
    "title": "Data Structures",
    "overview": "This course covers the fundamental concepts and applications of data structures in computer science. Students will explore various data structures such as arrays, linked lists, stacks, queues, trees, and graphs. The course will emphasize both the theoretical and practical aspects of data structures, including their implementation and analysis of their efficiency. By the end of the course, students will be equipped with the knowledge to use data structures effectively in real-world applications and advanced computing challenges.",
    "objectives": [
      "Understand the fundamental concepts of data structures, including their definition, representation, and operations.",
      "Apply data structures to solve real-world problems in various domains.",
      "Analyze the efficiency of data structures in terms of time and space complexity.",
      "Implement data structures in Python and evaluate their performance.",
      "Develop a deep understanding of the strengths and weaknesses of different data structures."
    ],
    "policies_and_exceptions": {
      "attendance_requirements": "Regular attendance is expected and absences must be communicated in advance. More than two unexcused absences may result in a deduction of your final grade.",
      "make-up_work": "Missed assignments and exams must be made up within one week of the original due date. Make-up exams may be different from the original exam.",
      "academic_honesty": "All work submitted for this course must be original and your own. Any instances of plagiarism will result in severe penalties, including failing the course."
    },
    "grade_level_assessments": {
      "assessment_components": {
        "assignments": 20,
        "exams": 30,
        "projects": 30,
        "participation": 10,
        "quizzes": 10
      },
      "grade_scale": {
        "A": "90-100%",
        "B": "80-89%",
        "C": "70-79%",
        "D": "60-69%",
        "F": "Below 60%"
      },
      "performance_expectations": "Students are expected to demonstrate a high level of understanding of the course material, participate actively in class discussions, and complete all assignments and projects on time.",
      "feedback_and_improvement": "Feedback on assignments and exams will be provided within one week of submission. Students are encouraged to seek additional feedback during office hours or by appointment."
    },
    "required_materials": {
      "recommended_books": [
        "Data Structures and Algorithms in Python",
        "Data Structures and Algorithms Made Easy",
        "Introduction to Algorithms"
      ],
      "required_items": [
        "Laptop with Python installed",
        "Notebook and pens",
        "Access to online learning platform"
      ]
    },
    "additional_information": {
      "prerequisites": "Basic programming skills in Python",
      "office_hours": "Mondays and Wednesdays, 2-3 PM",
      "contact_information": "Email: username@example.edu, Phone: 123-456-7890"
    }
  }
}

Test Coverage

Input Validation Tests:

• test_invalid_input: Ensures that invalid inputs such as an empty subject or missing grade level raise appropriate errors.

• test_valid_model and test_invalid_model: Validates the Pydantic model (SyllabusModel) to ensure that it correctly accepts valid data and raises errors for missing or incorrect fields.

Functionality Tests:

• test_create_prompt_temp_with_mocked_read_text_file: Checks if the create_prompt_temp function correctly assigns the appropriate grading policy based on the grade level, using mock inputs to simulate different school levels.

• test_compile: Verifies that the compile function correctly generates the prompt and model chain based on the type of compilation ("customization" or "syllabus").

• test_validate_response_valid and test_validate_response_invalid: Checks the validate_response method to ensure it correctly identifies valid and invalid syllabus structures.

How Tests Were Conducted

The tests were conducted using the pytest framework, which provides a robust environment for running test cases. This includes using fixtures for setting up test cases (sb fixture for the SyllabusBuilder instance) and mocking functions to simulate and control the behavior of dependencies.

Cases Covered

1. Input validation to ensure that the correct types and values are passed.

2. Prompt creation based on different grade levels to ensure the proper grading policies are applied.

3. Syllabus compilation to ensure that the model and prompt chains are correctly assembled and invoked.

4. Model validation to confirm that the syllabus structure adheres to the expected schema.

5. Response validation to make sure that the generated syllabus meets all necessary requirements.

Code Snippets

Input Validation Tests

test_invalid_input:

with pytest.raises(ValueError):
    SyllabusBuilder(subject='', grade_level=10)

with pytest.raises(ValueError):
    SyllabusBuilder(subject='Biology', grade_level=None)

test_valid_model and test_invalid_model:

valid_input = {
    'title': 'Sample Syllabus Title',
    'overview': 'Sample overview of the syllabus content.',
    'objectives': [
        'Define the key terms associated with...',
        'Describe the cardiac cycle...',
    ],
    'policies_and_exceptions': {
        'attendance_requirements': 'Sample attendance requirements...',
        'make_up_work': 'Sample make-up work policy...',
    },
    'grade_level_assessments': {
        'assessment_components': {
            'assignments': 20,
            'exams': 25,
            'projects': 25,
            'presentations': 15,
            'participation': 15,
        },
        'grade_scale': {
            'A': '90-100%',
            'B': '80-89%',
            'C': '70-79%',
            'D': '60-69%',
            'F': 'Below 60%',
        },
    },
    'required_materials': {
        'recommended_books': ['book 1', 'book 2', 'book 3'],
        'required_items': [
            'paint',
            'paintbrush',
            'pencil',
            'eraser',
            'notebooks', 'sharpies',
            'crayons',
            'black ink pen',
            'ruler',
        ],
    },
}

Functionality Tests

test_compile:

@patch.object(SyllabusBuilder, '\_create_custom_promptTemp')
@patch.object(SyllabusBuilder, '\_create_prompt_temp')
def test_compile(mock_create_prompt_temp, mock_create_custom_promptTemp):
    mock_custom_prompt_instance = MagicMock(spec=PromptTemplate)
    mock_create_custom_promptTemp.return_value = mock_custom_prompt_instance

    mock_syllabus_prompt_instance = MagicMock(spec=PromptTemplate)
    mock_create_prompt_temp.return_value = mock_syllabus_prompt_instance

    sb = SyllabusBuilder(subject='Mathematics', grade_level='Grade 5', verbose=True)

    chain_customisation = sb._compile('customisation')
    mock_create_custom_promptTemp.assert_called_once()
    chain_syllabus = sb._compile('syllabus')

    mock_create_prompt_temp.assert_called_once()

Testing documentation by @ElyasBinothman

Documentation made by team members

Syllabus Generator.pdf @YomnaEisa

[chitoroagad]

I watched the code review and I appreciate the feedback by the way :+1:.