search

ethz-spylab / satml-llm-ctf

Code used to run the platform for the LLM CTF colocated with SaTML 2024
https://ctf.spylab.ai
MIT License
25 stars 6 forks source link

Being among the first teams to break a defense #51

Closed NivC closed 10 months ago

NivC commented 10 months ago

Dear SpyLab,

There is something about the rules that seems a bit weird to me, I would appreciate if you could please clarify:

A team can get points by being the first team to discover a secret. I am guessing this refers to the attack stage. Yet, if a few teams break a certain defense during the reconnaissance phase, it essentially becomes a competition of who uses the API in the fastest way (hoping for no technical issues when the attack stage is first up).

Is my description accurate or am I missing something?

Thanks for maintaining the competitions! Niv.

dpaleka commented 10 months ago

Yes. This is supposed to be ameliorated by the decaying value of "easy" defenses, in case there are many attackers. If a defense is easy to the point that there will be a race of who uses the API first, it is to be expected that its value would decay later.

The rules are set, teams have been preparing, and at this point it's not fair to change the scoring mechanism signicantly.

The main thing that we can still change about this (before the Evaluation phase really takes off) is increase the factor \gamma. We wouldn't want to change anything about scoring after Monday, barring unforeseen technical issues.

NivC commented 10 months ago

Thanks for the quick response! I see how the factor can mitigate this issue.

Niv.