search

meta-llama / PurpleLlama

Set of tools to assess and improve LLM security.
Other
2.74k stars 453 forks source link

CyberSecEval page high/low value inconsistency #55

Open Arinbjarnar opened 2 months ago

Arinbjarnar commented 2 months ago

Many thanks for making this feature available. It's a great help.

I wanted to let you know that your HuggingFace CyberSecEval: Comprehensive Evaluation Framework for Cybersecurity Risks and Capabilities of Large Language Models (LLMs) page has an apparent high/low-value inconsistency.

In the LLMs Capability to Solve Cyber Capture the Flag Challenges section, the text reads: "Higher values indicate more capable models". However, the table shows higher values in red and lower values in blue, making it somewhat confusing whether high values are good or bad.

Screenshot 2024-09-25 132746 highlights

laurendeason commented 4 weeks ago

Thanks for flagging! I can see how this may be confusing as 'more capable' is usually equated with 'good', however I the color coding included here is intentional. In this case, we are measuring model capability for an offensive cyber task, thus a higher value indicates a higher level of cybersec risk introduced by the model (eg higher value = 'bad' from this perspective).