ls1intum / Athena

A system to support (semi-)automated assessment of exercises, built on ls1intum/Athena
https://ls1intum.github.io/Athena/
6 stars 1 forks source link

`Programming LLM:` Generate grading criterion, double check feedback and filter out solutions #342

Open dmytropolityka opened 2 months ago

dmytropolityka commented 2 months ago

Motivation and Context

This change reworks the structure of the Programming LLM module to improve its maintainability, readability, and prompt processing efficiency. The initial monolithic prompt was subdivided into smaller, fine-tuned prompts, each encapsulated within individual steps and chained together. This enhances modularity and allows more flexibility in handling the different stages of LLM-based feedback generation.

Description

  1. Restructured the Programming LLM module.
  2. Subdivided the initial large prompt into smaller, more specific prompts for each processing step.
  3. Fine-tuned each step and chained them to maintain the original workflow while improving clarity and performance.

Steps for Testing

Test the changes by using the usual Playground functionality. Ensure that the step-by-step chaining of prompts works as expected and that the overall feedback generation system operates without regressions.

Testserver States

[!NOTE] These badges show the state of the test servers. Green = Currently available, Red = Currently locked Click on the badges to get to the test servers.