ethz-spylab / satml-llm-ctf

Code used to run the platform for the LLM CTF colocated with SaTML 2024
https://ctf.spylab.ai
MIT License
25 stars 6 forks source link

Submitting secret #57

Closed jacoporepossi closed 9 months ago

jacoporepossi commented 9 months ago

Hello there,

I'm facing an issue where I submitted a correct secret using the interactive API docs (using the secret_guess) but when I check the scores endpoint I can't see my score correctly populated. Furthermore, retrying the scores endpoint gives me a 304 response status.

Thank you

dedeswim commented 9 months ago

Hi, the scores are cached and updated every minute to prevent the server from being overloaded and prioritize chat creation and secret guessing, so you have to wait at most 60 seconds to see your updated score.

dedeswim commented 9 months ago

I see that no scores from your team are not appearing yet. Are you sure that you're creating eval chats, as described in the API documentation and in the rules?

jacoporepossi commented 9 months ago

Actually, my bad. I was using the interface to chat with a given defence. Whenever I was able to get the secret, I was copying the secret_id and chat_id in the API docs and using the secret_guess endpoint to submit the attack.

I'm aware the interface is in Reconnaissance mode (so check the secret doesn't count), but I didn't know the whole chat system couldn't work the way I just described.

Sorry about that, I guess I'll have to code the attack even though the interface was quite handful.

dedeswim commented 9 months ago

We're sorry that we don't allow running evaluation phase stuff through the chat interface, but unfortunately, we believe that the UI library we are using for it is not reliable enough for this: it could send duplicate requests, or not send any at all, hence messing up with the numbers you would expect