[Question] restler_custom_payload. How to replace entire request body?

[Alucard23890]

Hi, I'm currently experimenting with restler and try to replace the entire request body of a resource with a custom payload. According to your documentation i tried to add this to my fuzzing dictionary and recompiled my project with the custom dictionary.

"restler_custom_payload": { "/api/movies/post/__body__": ["custom payload"] }

There exists an endpoint POST /api/movies to create a new resource. Problem is that restler during fuzz-lean or fuzzing modes never sends this body to the application.

Did I something wrong or did I misunderstand your documentation?

Thanks in Advance.

[marina-p]

Hello @Alucard23890,

Your custom payload above looks right. Do you see the key of the custom payload in grammar.py? If I add a similar custom payload when compiling the demo_server specification in the repo, I get the following:

"/blog/posts/post/__body__": ["custom payload"],

primitives.restler_custom_payload("/blog/posts/post/__body__", quoted=False),

The statement in the grammar should make RESTler send the custom payload instead of the body.

Thanks,

Marina

[Alucard23890]

Hello @marina-p it seems that my grammar.py doesn't contain this key at all. i tried to use this in my dict.json:

 "/api/movies/post/__body__": ["custom_body"],

or this (since /api is the basepath)

 "/movies/post/__body__": ["custom_body"],

but nothing works whereas

"length": ["01:30"],

works. length is a property of the json body.

I get this python code for my endpoint:

# Endpoint: /movies, method: Post
request = requests.Request([
    primitives.restler_static_string("POST "),
    primitives.restler_basepath("/api"),
    primitives.restler_static_string("/"),
    primitives.restler_static_string("movies"),
    primitives.restler_static_string(" HTTP/1.1\r\n"),
    primitives.restler_static_string("Accept: application/json\r\n"),
    primitives.restler_static_string("Host: localhost\r\n"),
    primitives.restler_static_string("Content-Type: "),
    primitives.restler_static_string("application/json"),
    primitives.restler_static_string("\r\n"),
    primitives.restler_refreshable_authentication_token("authentication_token_tag"),
    primitives.restler_static_string("\r\n"),
    primitives.restler_static_string("{"),
    primitives.restler_static_string("""
    "title":"""),
    primitives.restler_fuzzable_string("fuzzstring", quoted=True),
    primitives.restler_static_string(""",
    "description":"""),
    primitives.restler_fuzzable_string("fuzzstring", quoted=True),
    primitives.restler_static_string(""",
    "length":"""),
    primitives.restler_fuzzable_string("fuzzstring", quoted=True),
    primitives.restler_static_string("}"),
    primitives.restler_static_string("\r\n"),

    {

        'post_send':
        {
            'parser': parse_moviespost,
            'dependencies':
            [
                _movies_post_data_id.writer()
            ]
        }

    },

],
requestId="/movies"
)

the related swagger yaml part looks like this:

post:
  tags:
    - Movies
  requestBody:
    required: true
    content:
      application/json:
        schema:
          $ref: '#/components/schemas/MovieBody'
  responses:
    201:
      description: "Movie created"
      content:
        application/json:
          schema:
            type: object
            properties:
              data:
                $ref: '#/components/schemas/Movie'
    400:
      description: "Bad Input"

Could it be that the problem is how i specified the request body in swagger?

[marina-p]

Hello @Alucard23890,

The body you have specified is working for me with the following custom payload without the base path:

 "/movies/post/__body__": ["custom_body"],

Could you please double check you are referencing the dictionary with this payload in your compiler config? In your grammar above, there are no custom payloads, which could be because the dictionary to which you have added new payloads is not referenced.

Thanks,

Marina

[Alucard23890]

Hello @marina-p yes i double checked it and the dict_0.json which is generated in the compile folder contains my changes but still no custom payload in grammar.py. The custom payload is working when i set it only for a single property like length as mentioned in my previous comment. only the body is not working.

I use this json for my compile config:

{
    "SwaggerSpecConfig": [
        {

            "DictionaryFilePath": "<absolute-path>/dict.json",

            "SpecFilePath": "<absolute-path>/openapi.yaml"
        }
    ]
}

I'm working on a macbook. could this be a problem? until now, everything worked without problems.

[Alucard23890]

Hi @marina-p , I tried now many different things, compiled restler on a windows machine with .NET SDK 6 used the newest commit of this repo, convert my openapi documentation to swagger2 (was OpenAPI 3.0) but still no success. Now I can't even reproduce that restler uses the custom payload of property length, which worked at the beginning. But restler still copies my dictionary to a file dict_0.json inside the Compile folder.

I uploaded you an ZIP File with all used files (compiler config, the swagger doc and my custom dictionary) and the generated grammar.py from this files. I really don't see what I am doing wrong. Archive.zip

[marina-p]

Hello @Alucard23890,

Thank you for the repro files, these were very helpful in diagnosing the issue. The problem reproduces only when using SwaggerSpecConfig, and the custom payload is missing due to a bug in RESTler.

We will fix this ASAP. In the meantime, the only workaround to get past this issue is to use the global config parameters CustomDictionaryFilePath and SwaggerSpecFilePath.

Also, while working on this, I saw another problem, which is that the default settings are not being applied if you only include the Swagger and dictionary path (I will open a separate issue on this). To work around this, please do the following:

1. Compile the spec using restler compile --api_spec <spec>.
2. Copy the generated config.json from the Compile folder.
3. Replace with your paths to the dictionary and specification.

Thanks,

Marina

[Alucard23890]

Hi @marina-p Thank you very much for your help. With your latest suggestion I finally managed to delivery my custom payload to my application.

[CyberJhin]

Hi @marina-p Thank you very much for your help. With your latest suggestion I finally managed to delivery my custom payload to my application.

Hello @Alucard23890, could you show me what the inserted body looks like? I should have a huge json object in the body with a lot of attachments, but if I insert it into ""/documents/post/body": ["custom_body"]" instead of "custom_body" I get a lot of syntax errors

[Alucard23890]

Hi @KappaHacker if you want to deliver a custom body to your api the key should look like this according to your example:

{
  "/documents/post/__body__": ["<write here your json>"]
}

you have to recompile it after you made changes to your custom dictionary.

[marina-p]

Hello @KappaHacker,

Did you escape the json prior to embedding it as a string? If not, that could be the reason for your syntax errors. For example:

"tags": [
    "{\"key1\": \"value1\"}"
],

Another approach, which may work better for you, is to specify the body through an example payload (see Examples.md). Then, you can simply add the body to the example as an object as in the snippet below.

"parameters": {
   "__body__": {<your object here> }
}

Thanks,

Marina

[CyberJhin]

@marina-p, I don't understand how to use the examples... I created a folder where I put the json file with the example. In config.json I specified DiscoverExamples: true and ExamplesDirectory: "the path to the folder with the example", But what should I do next? Where should I change yet? In swagger? Test.zip

[marina-p]

Hello @KappaHacker,

The missing piece is that you need to link the endpoint and method to your example. This can be done in one of two ways: from within the Swagger, or in a separate file.

Here are the steps that do not involve changing your Swagger file:

• create a new json file that has the endpoint, method, and path to your example file, or simply specify the example inline in this file - see Examples.md section "Specifying examples in a separate file" for more details showing a sample for specifying a file or inline example. The config file will look like:

  {
  "paths": {
      "/blog/posts": {
         "post": {
             "first example":    {                 
                 "parameters": {
                      "__body__": {}
                  }
              }
         }
      }
  }
  }

• in config.json, set ExampleConfigFilePath to the path of the above json file.

Does this clarify how to get the example for the body plugged in?

Thanks,

Marina