search

autolab / Autolab

Course management service that enables auto-graded programming assignments.
http://www.autolabproject.com/
Apache License 2.0
743 stars 209 forks source link

Add detection for mismatch in Autograder results and assessment problems #2112

Closed evanyeyeye closed 3 months ago

evanyeyeye commented 4 months ago

Description

image image

Motivation and Context

Closes #2087.

How Has This Been Tested?

Types of changes

coderabbitai[bot] commented 4 months ago
Walkthrough ## Walkthrough The changes address the issue of mismatches between Autograder results and assessment problems by introducing functionality to detect and report missing problems. This includes recording missing problems during autograding, updating the database schema to support this data, and adjusting the UI to notify users of any discrepancies. The adjustments ensure that missing problems are clearly identified and communicated, improving the transparency and usability of the autograding feedback process. ## Changes | File Path | Changes | |-----------|---------| | `app/helpers/.../assessment_autograde_core.rb` | Introduced handling for missing problems during autograding and adjusted logic for setting problem scores. | | `app/views/assessments/.../_submission_history_row.html.erb` | Added UI logic to display a message for missing problems in autograder results. | | `db/migrate/.../20240226194217_add_missing_problems_to_submissions.rb` | Added a migration to include a `missing_problems` text column in the `submissions` table. | | `db/schema.rb` | Updated schema to reflect the addition of the `missing_problems` column in the `submissions` table. | ## Assessment against linked issues | Objective | Addressed | Explanation | |-----------|-----------|-------------| | Better detection of mismatch in Autograder results and assessment problems (#2087) | ✅ | | | Notification for instructors about missing problems in Autograder results (#2087) | ✅ | | The changes effectively address the objectives outlined in issue #2087 by implementing mechanisms for better detection and reporting of mismatches between Autograder results and assessment problems. The introduction of a `missing_problems` field in the database, along with the UI enhancements for notifying instructors, directly tackles the problem of unreported mismatches, fulfilling the requirements of the linked issue without any apparent omissions.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share - [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)
Tips ### Chat There are 3 ways to chat with CodeRabbit: - Review comments: Directly reply to a review comment made by CodeRabbit. Example: - `I pushed a fix in commit .` - `Generate unit-tests for this file.` - `Open a follow-up GitHub issue for this discussion.` - Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples: - `@coderabbitai generate unit tests for this file.` - `@coderabbitai modularize this function.` - PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples: - `@coderabbitai generate interesting stats about this repository and render them as a table.` - `@coderabbitai show all the console.log statements in this repository.` - `@coderabbitai read src/utils.ts and generate unit tests.` - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.` Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. ### CodeRabbit Commands (invoked as PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger a review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai resolve` resolve all the CodeRabbit review comments. - `@coderabbitai help` to get help. Additionally, you can add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. ### CodeRabbit Configration File (`.coderabbit.yaml`) - You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository. - The JSON schema for the configuration file is available [here](https://coderabbit.ai/integrations/coderabbit-overrides.v2.json). - If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json` ### CodeRabbit Discord Community Join our [Discord Community](https://discord.com/invite/GsXnASn26c) to get help, request features, and share feedback.