The LLMM Verification Plugin ("LLMV") is a CTFd plugin that adds a new challenge type called "LLM Verification." This new challenge type tasks users with creating prompts for external LLM APIs (such as gpt-neox-20b) that generate cheeky responses. These responses are reviewed manually by graders, who assign points based on how successfully the answer managed to subvert the model.'
This plugin is should be used wth LLM Router which provides routes the requests to the configured LLM providers.
This is the suggested approach. Instructions Coming soon
Copy the llm_verification
folder that contains this README.md
file to CTFd/plugins/
.
Copy the template config file (llmv_config.template.json
) from CTFd/plugins/llm_verification/ and remove
.template` from the filename.
$ cp CTFd/plugins/llm_verification/llmv_config.template.json CTFd/plugins/llm_verification/llmv_config.json
Replace "UNSET"
values llmv_config.json
with the values that you desire.
Confirm that LLMV installed successfully. Look for this message after running docker compose up
.
INFO - Initialized LLM Verification Plugin
Create a CTFd event.
Login to CTFd as an administrator.
Click `Admin Panel" on the right side of the top toolbar.
Click "Challenges" in middle of the top toolbar.
Click the ⨁
in the "Challenges" page's title (Challenges ⨁
).
Create a challenge with "llm_verification" selected as the "Challenge Type."
a. Name (optional): challenge title.
b. Category (optional): general "type" of challenge that's used to visually group challenges.
c. Message (optional): Describe the goal for your users.
d. Pre-prompt (optional): A string that will be prepended to every prompt that users submit to this challenge. Be sure to leave a space at the end.
e. LLM to use (optional, defaults to "VanillaNeox"): The LLM to use for this challenge.
f. Value (required): This isn't used, but you need to add an integer to proceed.
g. Click the "Create" button in the lower right, which will present you with an "Options" popup.
h. Flag (optional): This value isn't used and can be ignored.
i. State (optional): Set to "Visible" so users can see your new challenge.
Click "CTFd" on the left of the top toolbar. This exits the "Admin Panel."
Click "Challenges" in the middle of the top toolbar. This shows challenges as users see them.
Select the challenge that you created.
Add some text to the "Prompt" text box.
Click the "Generate" button. Generated text will appear in the second box. If you'd like to change the text that was generated, then change the prompt and click "Generate" again. Previously generated text in the second box will be replaced.
Click the "Submit" button. A red notification will pop up saying "Submission Under Review."
Click "Admin Panel" in the right of the top panel.
Click the "Plugins" dropdown in the right of the top panel.
Click "LLM Submissions" to navigate to the grading page for answer submissions.
Click the 💬 ("Grade") button on the far right of an answer submission's row.
Click "Mark Incorrect," "Mark Correct," or "Award Points." a. 🟥 "Mark Incorrect:" Delete the user's submission and don't award them points. b. 🟩 "Mark Correct:" Award the amount of points set earlier in "Options" and prevent the user from submitting additional answers. The challenge's card will be turn green and gain a ✅. The user will still be able to generate responses for the challenge, but clicking the "Submit" button will create a blue dialog that says "You already solved this." c. ◻️"Award Points:" Award a custom amount of points and allow the user to submit additional answers. d. Note that points awarded with "Award Points" won't immediately show up in "Admin Panel"'s "Scoreboard", but they will show up immediately in the non-admin panel's "Scoreboard"
Use docker-compose.dev.yml
for making code changes to CTFd or plugins. Add --build
for non-code changes such as dependency changes in requirements.txt
or entrypoint changes in Dockerfile
. CTFd will be available at https://localhost:8000.
Since the repo directory's already mounted as a volume in docker-compose.yml
, container images don't need to be rebuilt for code changes to take effect. However, (the production server) Gunicorn won't recognize these changes, so we need to start it with Flask instead.
$ docker compose -f docker-compose.dev.yml up --build
To hack on the plugin's code, we want to start Flask (CTFd's RESTful) in development mode so code changes trigger hot reloads.
The Dockerfile.dev
, docker-compose.dev.yml
, and docker-entrypoint.dev.sh
are near-exact copies of Dockerfile
, docker-compose.yml
, and docker-entrypoint.sh
. Dockerfile.dev
uses a different entrypoint (docker-entrypoint.dev.sh
) and docker-compose.dev.yml
uses a different Dockerfile for builds. They work together to start the application with Flask instead of Gunicorn (via docker-entrypoint.dev.sh
).
To use it, specify it with -f
when invoking docker compose up
.
$ docker compose -f docker-compose.dev.yml up
For dependency and Dockerfile changes to take effect, add --build
. This will take longer, though, because some layers will need to be rebuilt.
$ docker compose -f docker-compose.dev.yml up --build
To start over from a clean slate, ensure that no CTFd containers are running with docker ps
and docker kill
. Then run rm -rf ./.data
.
(Presumably) Compatible with CTFd v3.0.0
Built with CTFd v3.5.1
If you experience any problems, or if you think you've found a bug, or have a feature request - please don't hesitate to reach out to support@ctfd.io.
Readme format inspired by makeareadme.com.
Must be 16 or older and have an adult in the car during operation.