REST URI client error: 422

Summary

Hi all,

i'm trying to get garak to run with the rest generator on an llm project i'm in atm. Whatever i do, i always get 422. been testing all the id's, token and post/get requests without problems. maybe someone has any idea on what i'm doing wrong. i will put some copied info out of my shell (redacted of course)

Relevant information

username@xy:~$ garak --model_type rest -G jsonfile.json --model_name "<uri>"

garak LLM security probe v0.9.0.11.post1 ( https://github.com/leondz/garak ) at xy

📜 reporting to runs/garak.xy.report.jsonl

🦜 loading generator: REST: <uri>

🕵️  queue of probes: atkgen.Tox, continuation.ContinueSlursReclaimedSlurs80, dan.AntiDAN, dan.AutoDANProbe, dan.ChatGPT_Developer_Mode_RANTI, dan.ChatGPT_Developer_Mode_v2, dan.ChatGPT_Image_Markdown, dan.DAN_Jailbreak, dan.DUDE, dan.Dan_10_0, dan.Dan_11_0, dan.Dan_6_0, dan.Dan_6_2, dan.Dan_7_0, dan.Dan_8_0, dan.Dan_9_0, dan.STAN, encoding.InjectAscii85, encoding.InjectBase16, encoding.InjectBase2048, encoding.InjectBase32, encoding.InjectBase64, encoding.InjectBraille, encoding.InjectHex, encoding.InjectMorse, encoding.InjectROT13, encoding.InjectUU, gcg.GCG_Probe, glitch.Glitch100, goodside.Davidjl, goodside.Tag, goodside.ThreatenJSON, goodside.WhoIsRiley, knownbadsignatures.EICAR, knownbadsignatures.GTUBE, knownbadsignatures.GTphish, leakreplay.GuardianCloze, leakreplay.GuardianComplete, leakreplay.LiteratureCloze80, leakreplay.LiteratureComplete80, leakreplay.NYTCloze, leakreplay.NYTComplete, lmrc.Anthropomorphisation, lmrc.Bullying, lmrc.Deadnaming, lmrc.Profanity, lmrc.QuackMedicine, lmrc.SexualContent, lmrc.Sexualisation, lmrc.SlurUsage, malwaregen.Evasion, malwaregen.Payload, malwaregen.SubFunctions, malwaregen.TopLevel, misleading.FalseAssertion50, packagehallucination.Python, promptinject.HijackHateHumansMini, promptinject.HijackKillHumansMini, promptinject.HijackLongPromptMini, realtoxicityprompts.RTPBlank, realtoxicityprompts.RTPFlirtation, realtoxicityprompts.RTPIdentity_Attack, realtoxicityprompts.RTPInsult, realtoxicityprompts.RTPProfanity, realtoxicityprompts.RTPSevere_Toxicity, realtoxicityprompts.RTPSexually_Explicit, realtoxicityprompts.RTPThreat, replay.Repeat, snowball.GraphConnectivityMini, snowball.PrimesMini, snowball.SenatorsMini, xss.MarkdownImageExfil

/home/username/.local/lib/python3.9/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884

  warnings.warn(

🔴🪖  🦜 loading generator: Hugging Face 🤗 pipeline: leondz/artgpt2tox

Traceback (most recent call last):

  File "/home/username/.local/bin/garak", line 8, in <module>                                                                                                                                              | 1/10 [00:05<00:49,  5.47s/it]

    sys.exit(main())

  File "/home/username/.local/lib/python3.9/site-packages/garak/__main__.py", line 9, in main

    cli.main(sys.argv[1:])

  File "/home/username/.local/lib/python3.9/site-packages/garak/cli.py", line 475, in main

    command.probewise_run(generator, probe_names, evaluator, buffs)

  File "/home/username/.local/lib/python3.9/site-packages/garak/command.py", line 212, in probewise_run

    probewise_h.run(generator, probe_names, evaluator, buffs)

  File "/home/username/.local/lib/python3.9/site-packages/garak/harnesses/probewise.py", line 108, in run

    h.run(model, [probe], detectors, evaluator, announce_probe=False)

  File "/home/username/.local/lib/python3.9/site-packages/garak/harnesses/base.py", line 92, in run

    attempt_results = probe.probe(model)

  File "/home/username/.local/lib/python3.9/site-packages/garak/probes/atkgen.py", line 133, in probe

    response = generator.generate(challenge)[0].strip()

  File "/home/username/.local/lib/python3.9/site-packages/garak/generators/base.py", line 78, in generate

    return [self._call_model(prompt)]

  File "/home/username/.local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry

    ret = target(*args, **kwargs)

  File "/home/username/.local/lib/python3.9/site-packages/garak/generators/rest.py", line 245, in _call_model

    raise ConnectionError(

ConnectionError: REST URI client error: 422 - Unprocessable Entity

This is how my jsonfile.json looks like:

{

  "RestGenerator": {

    "name": "somename",

    "uri": "<right uri>",

    "method": "post",

    "headers": {

      "Authorization": "Bearer <functioning token>",

      "Accept": "application/json",

      "Content-Type": "application/json"

    },

    "req_template_json_object": {

      "payload": "$INPUT",

      "origin": "user",

      "channel_id": "<functioning id>",

      "conversation_id": "<functioning id>"

    },

    "response_json": true,

    "response_json_field": "message.payload",

    "request_timeout": 20,

    "ratelimit_codes": [429]

  }

}

Basic Rest troubleshooting steps I've used. Try curl with the desired json data and headers. Use test.Test probe until you have your Rest generator config working. Add debug lines to rest.py to see what the data looks like (copy below commented sections and uncomment) If the llm app is poorly designed and returning HTTP errors for non-API issues, then you might need to capture the errors and continue (see below). I get this with with some LLM Firewalls as well as an internal app that has a very limited input length. Below is my modifications to put errors into the response. From there you could have a detector looking for the errors and adding that to the detector list.

Code from my hacked Rest generator.(1)

        req_kArgs = {
            data_kw: request_data,
            "headers": request_headers,
            "timeout": self.request_timeout,
        }
#        logging.debug(  "RestGenerator Request content: %s"
#                        % repr(req_kArgs)
#                        )

        resp = self.http_function(self.uri, **req_kArgs)

#        logging.debug(  "RestGenerator Response content: %s"
#                        % repr(resp.content)
#                        )
        if resp.status_code in self.ratelimit_codes:
            response = [f"REST URI Rate limited: {resp.status_code} - {resp.reason}"]

        elif str(resp.status_code)[0] == "3":
            response = [f"REST URI redirection: {resp.status_code} - {resp.reason} \n {resp.content}"]

        elif str(resp.status_code)[0] == "4":
            response = [f"REST URI client error: {resp.status_code} - {resp.reason} \n {resp.content}"]

        elif str(resp.status_code)[0] == "5":
            response = [f"REST URI server error: {resp.status_code} - {resp.reason} \n {resp.content}"]

        elif not self.response_json:
            return [str(resp.text)]

(1) Note to my employer: I'm helping someone out, not contributing code to an open source project.

leondz / garak

REST URI client error: 422 #952

Summary

Relevant information