ethz-spylab / satml-llm-ctf

Code used to run the platform for the LLM CTF colocated with SaTML 2024
https://ctf.spylab.ai
MIT License
23 stars 5 forks source link

Is the secret alphanumeric or not? #29

Closed epistoteles closed 8 months ago

epistoteles commented 8 months ago

In the rules, the secret is described as alphanumeric.

However, in the API docs, the secret is also instantiated to be something else, e.g. 's]]]]]':

For me alphanumeric means a-zA-Z0-9, and nothing else. This is important for my Python filter. Could you please clarify?

Jordine commented 8 months ago

Also, will the secrets all be strictly 6 characters long? I can't set it to anything longer in the interface.

epistoteles commented 8 months ago

Edit: It looks like this is some serious failure in the backend. No matter what secret I put in the chat API, the secret is always 's]]]]]' for the model and all filters.

@Jordine: The schema for the secret in the API is 6 characters, matches [[:ascii:]]{6}.

dedeswim commented 8 months ago

Hi @epistoteles can you please tell me the steps to reproduce this? Where do you see what the secret is once the conversation is instantiated?

epistoteles commented 8 months ago

I'm not able to reproduce the error anymore, I'm really not sure what caused it. I'll consider this fixed for now or an error of mine.

dedeswim commented 8 months ago

Hi thanks for reporting that you're not having the issue anymore. Just so you know, we didn't make any changes on our end.

As per your question from above, the secret will be composed of exactly 6 alphanumeric characters.

KrystofM commented 8 months ago

I am having the same issue in the example of the secret within the SwaggerUI, only generating strings with first character being random and the rest being ]]]]]:

Screenshot 2024-01-17 at 11 00 51 AM

This could be just a problem of Swagger, however. Anyway it would nice to specify alphanumeric properly as I don't think the definition is stable; for me it meant english only uppercase A-Z (26) and 0-9 (10). In case is it either [a-zA-Z0-9] or [A-Z0-9] it would be nice to change that specification in the OpenAPI documentation, instead of leaving [[::ascii::]]{6}, which is a much larger set of characters. I would also suggest possibly leaving out the minLength and maxLength fields and instead just using the start and endline regex tokens ^$ to have an exact match.

{
      "type":"string",
      "pattern":"^[a-zA-Z0-9]{6}$",
      "title":"Secret"
}

or

{
      "type":"string",
      "pattern":"^[A-Z0-9]{6}$",
      "title":"Secret"
}
dedeswim commented 8 months ago

Hi, we'll change the OpenAPI specification ASAP to make sure things are clear.

In any case, we confirm that the secrets are going to be only alphanumeric characters, where the letters can be both lowercase and uppercase.

dedeswim commented 8 months ago

Done!