Generating request dependencies without an POST response body

[samuelspagl]

This is more like a question, or possible enhancement feature. But first of all, I thanks for this cool toolkit, reading your paper was very interesting!

In my working case we have a REST API that return a "201"-statuscode and nothing in the response body for a POST request. Therefore, if I read your paper correctly, Restler is not able to infer a dependency between a "POST api/stuff" and a "GET api/stuff/{stuffID}".

As the "stuffID" is required in the POST request body, I wanted to ask if it is possible to establish a dependency that's using the "stuffID" generated for the request and saves it for the following GET request, not using a response body.

I hope you kinda understand what I mean. 😅

Anyway, thanks a lot in advance. Best Samuel

[marina-p]

Hello @samuelspagl,

Yes, if it's specified in the POST request body, it should be possible to declare a dependency for it via the annotation file (see Annotations.md), which you can reference in the compiler config (see CompilerConfig.md). (This is a relatively new feature, which was not implemented at time of publication of the initial RESTler paper.)

In the annotation file, you would have something like the below (assuming stuffID in the body is a top-level property):

{
  "x-restler-global-annotations": [
    {
      "producer_endpoint": "/api/stuff",
      "producer_method": "POST",
      "producer_resource_name": "/stuffID",
      "consumer_param": "stuffID"
    }
  ]
}

Please let us know if the above is unclear or if you run into any issues.

Thanks,

Marina

[samuelspagl]

Hey @marina-p Ah sorry then I got something mixed up in my mind :D I read so many things about Restler the last few days :D

So first thanks, the compiling process works as expected. The dependencies seem to be correctly listed in the grammar.py. Despite that, when testing the new grammar, it's inserting null and not the right value.

2022-01-12 12:25:43.392: Sending: 'POST /v1/parameters HTTP/1.1\r\nAccept: application/json\r\nHost: \r\nContent-Type: application/json\r\nContent-Length: 101\r\nUser-Agent: restler/8.3.0\r\n\r\n{\n    "id":"50f2117d-d34f-461e-81a9-d7857e81daf4",\n    "name":"fuzzstring",\n    "tag":"fuzzstring"}\r\n'

2022-01-12 12:25:43.400: Received: 'HTTP/1.1 201 Created\r\ncontent-length: 0\r\n\r\n'

2022-01-12 12:25:43.411: Sending: 'GET /v1/parameters?parameterId=null HTTP/1.1\r\nAccept: application/json\r\nHost: \r\nParameter-Header: fuzzstring\r\nContent-Length: 0\r\nUser-Agent: restler/8.3.0\r\n\r\n'

2022-01-12 12:25:43.421: Received: 'HTTP/1.1 400 Bad Request\r\ncontent-type: application/json\r\ncontent-length: 151\r\n\r\n{"message":"[Bad Request] Validation error for parameter parameterId in location QUERY: Provided value don\'t match pattern","code":400,"subsystem":999}'

The grammar file excerpt in this example:

request = requests.Request([
    primitives.restler_static_string("POST "),
    primitives.restler_static_string("/"),
    primitives.restler_static_string("v1"),
    primitives.restler_static_string("/"),
    primitives.restler_static_string("parameters"),
    primitives.restler_static_string(" HTTP/1.1\r\n"),
    primitives.restler_static_string("Accept: application/json\r\n"),
    primitives.restler_static_string("Host: \r\n"),
    primitives.restler_static_string("Content-Type: "),
    primitives.restler_static_string("application/json"),
    primitives.restler_static_string("\r\n"),
    primitives.restler_refreshable_authentication_token("authentication_token_tag"),
    primitives.restler_static_string("\r\n"),
    primitives.restler_static_string("{"),
    primitives.restler_static_string("""
    "id":"""),
    primitives.restler_fuzzable_uuid4("566048da-ed19-4cd3-8e0a-b7e0e1ec4d72", quoted=True, param_name="id"),
    primitives.restler_static_string(""",
    "name":"""),
    primitives.restler_fuzzable_string("fuzzstring", quoted=True, param_name="name"),
    primitives.restler_static_string(""",
    "tag":"""),
    primitives.restler_fuzzable_string("fuzzstring", quoted=True, param_name="tag"),
    primitives.restler_static_string("}"),
    primitives.restler_static_string("\r\n"),

    {

        'post_send':
        {

            'dependencies':
            [
                _v1_parameters_post_id.writer()
            ]
        }

    },

],
requestId="/v1/parameters"
)
req_collection.add_request(request)

# Endpoint: /v1/parameters/{parameterId}, method: Get
request = requests.Request([
    primitives.restler_static_string("GET "),
    primitives.restler_static_string("/"),
    primitives.restler_static_string("v1"),
    primitives.restler_static_string("/"),
    primitives.restler_static_string("parameters"),
    primitives.restler_static_string("/"),
    primitives.restler_static_string(_v1_parameters_post_id.reader(), quoted=False),
    primitives.restler_static_string(" HTTP/1.1\r\n"),
    primitives.restler_static_string("Accept: application/json\r\n"),
    primitives.restler_static_string("Host: \r\n"),
    primitives.restler_refreshable_authentication_token("authentication_token_tag"),
    primitives.restler_static_string("\r\n"),

],
requestId="/v1/parameters/{parameterId}"
)
req_collection.add_request(request)

[samuelspagl]

@marina-p Additionally this is reproducible in the demo_server by deleting the response body schema in the POST /api/blog/posts request, and adding the global annotations via an annotations.json to the compiler settings. By doing this, the rendering sequence is not correctly assigning the "postId".

I hope this is helpful :)

[marina-p]

@samuelspagl,

Thank you for the details. There is a bug in the grammar - will investigate (looks like a bug in RESTler).

In the below grammar element, there should be another parameter to save the value, but it is missing:

    primitives.restler_fuzzable_uuid4("566048da-ed19-4cd3-8e0a-b7e0e1ec4d72", quoted=True, param_name="id", writer=_v1_parameters_post_id.writer())),

[samuelspagl]

@marina-p Thanks for your response. I tried to manually add the writer parameter, but this doesn't change the fact, that "null" is still being inserted.

I know its kinda selfish to ask, but do you know, when you/your team will have time to look into a possible fix? Just that I can kinda plan the next steps, as this delays utilising Restler for myself.

Thanks a lot :)

[marina-p]

@samuelspagl,

This should be fixed very soon.

Just FYI, the 'writer' variable is not supported in every grammar element at this time. This is why the workaround to just add the 'writer' in the grammar does not resolve the issue.

Thanks,

Marina

[marina-p]

Hello, just FYI - this is being worked on, but the fix turned out to be quite involved. We should have another update toward the end of this week.

[samuelspagl]

@marina-p Thanks a lot for the update, I really appreciate it!

Really looking forward. :)

[marina-p]

Hello @samuelspagl,

If you get a chance, could you please try the draft version available in the 'mp/440_all' branch and let us know if this resolves your real use case?

I have confirmed that this branch fixes the demo_server repro and the restler_fuzzable_uuid case discussed above.

Thanks,

Marina

[samuelspagl]

Hey @marina-p So I tried to compile new binaries with the branch you mentioned, and it seems that for some reason the "engine" folder failed to be build. I tried this with my old repository fetching the new branches and using checkout to switch to the 'mp/440_all' branch on a Linux System, as well as cloning the repository completely fresh, switching the branch, and trying to build the binaries.

Still im going to try using the old engines binaries with the new compiler and restler ones.

[samuelspagl]

So compiling the openapi spec with my separate annotations file seems to be correct now, it is creating the writer=_v1_things_post_id.writer() in the primitives.restler_fuzzable_uuid4 and also references in the end as a dependency.

I'm now setting up my test environment again to see if it also works in RESTler test mode.

I keep you updated. :)

[samuelspagl]

Okay so I tried to start RESTler with the old engine binaries, but that fails directly.

[marina-p]

Thank you for trying out the changes. What was the engine build error? (That is expected, both compiler and engine changes are needed to make this work.)

[samuelspagl]

Hey @marina-p,

no problem, I'm happy to help. This is the screenshot of the building process using the build script provided in your repository.

Like I mentioned, it only generates the compiler, restler, and resultsAnalyzer binaries, but not the ones of the engine.

There is no real error message 😅.

Using the old engine binaries (that worked before with the other old binaries ^^) results in this error:

Looking into the EngineStdOut.txt the error is described as followed: `Cannot import grammar: module 'engine.primitives' has no attribute 'restler_basepath'

So I don't know if you also changed something in the engine, but yeah that said, is there any way to get more debug information of the building process? So that I could provide you with more information?

[marina-p]

Hi @samuelspagl,

Thank you, this is helpful. We'll need to improve the build script to help diagnose this.

I went ahead and merged the changes. Could you please try to pull the latest from the main branch, and let me know if you see the build error on the main branch as well?

Thanks,

Marina

[samuelspagl]

Hey @marina-p

So no it doesn't work, I made a fresh clone in a completely new folder on both a Linux (VM UTM) and my MacOS. Linux runs on Dotnet 5.0, my Mac uses Dotnet 6.0. With Dotnet 6.0 it is not possible to run RESTler, but with the previous versions it was always possible to compile the binaries.

[marina-p]

Hello @samuelspagl,

I just merged a change to add the build output that shows why the engine compilation failed. Could you please update and re-run the build script and share the additional output?

Thanks,

Marina

[marina-p]

@samuelspagl See related discussion in #454 - it is possible you are hitting the same issue.

[samuelspagl]

Hey @marina-p sorry for the late response. So yeah the argument --python_path /usr/bin/python3 helped to build everything just fine. In terms of MacOS it's also maybe to be noted that, when using an M1 Mac and Homebrew to install python the path differs a bit ;)

But now to the main part. So basically it seems that the relation between the requests is now interpreted correctly. But I have another problem:

This is from the logs:

2022-02-09 08:50:44.951: Sending: 'POST //v1/parameters HTTP/1.1\r\nAccept: application/json\r\nHost: \r\nContent-Type: application/json\r\nContent-Length: 101\r\nUser-Agent: restler/8.3.0\r\n\r\n{\n    "id":"6fdd67f1-ddf2-4a91-9bb4-8954c4484272",\n    "name":"fuzzstring",\n    "tag":"fuzzstring"}\r\n'

2022-02-09 08:50:44.954: Received: 'HTTP/1.1 201 Created\r\ncontent-length: 0\r\n\r\n'

2022-02-09 08:50:44.956: Sending: 'GET //v1/parameters/\\"6fdd67f1-ddf2-4a91-9bb4-8954c4484272\\" HTTP/1.1\r\nAccept: application/json\r\nHost: \r\nContent-Length: 0\r\nUser-Agent: restler/8.3.0\r\n\r\n'

2022-02-09 08:50:44.958: Received: 'HTTP/1.1 400 Bad Request\r\ncontent-type: application/json\r\ncontent-length: 150\r\n\r\n{"message":"[Bad Request] Validation error for parameter parameterId in location PATH: Provided value don\'t match pattern","code":400,"subsystem":999}'

As it can be seen, the POST request works just fine, but afterwards the GET requests fails. I'm pretty sure that this is due to the quotes in which the uuid in the request string is covered. In my compiled grammar its marked as not being covered in quotes:

# Endpoint: /v1/parameters/{parameterId}, method: Get
request = requests.Request([
    primitives.restler_static_string("GET "),
    primitives.restler_basepath("/"),
    primitives.restler_static_string("/"),
    primitives.restler_static_string("v1"),
    primitives.restler_static_string("/"),
    primitives.restler_static_string("parameters"),
    primitives.restler_static_string("/"),
    primitives.restler_static_string(_v1_parameters_post_id.reader(), quoted=False),
    primitives.restler_static_string(" HTTP/1.1\r\n"),
    primitives.restler_static_string("Accept: application/json\r\n"),
    primitives.restler_static_string("Host: \r\n"),
    primitives.restler_refreshable_authentication_token("authentication_token_tag"),
    primitives.restler_static_string("\r\n"),

],
requestId="/v1/parameters/{parameterId}"
)
req_collection.add_request(request)

The "POST" request has the parameter quoted for the UUID set to true as it is included in the json body. My question is, on what point is the is the UUID string saved, upon creation, or after the request body is built, including the quotes?

The GET request, needed to be valid would be GET //v1/parameters/6fdd67f1-ddf2-4a91-9bb4-8954c4484272

[marina-p]

Hello, @samuelspagl,

This was indeed a bug with saving the quoted string. I just merged a fix, could you please let me know if this resolves your issue?

Thanks,

Marina

[samuelspagl]

Hey @marina-p so yeah it works now.

So hopefully now one of the last questions to this issue topic :D. With the additional annotations.json it is possible to let RESTLer understand which requests are dependent. This works just fine now, however it's not path but name dependent right?

So in our case: In our openapi.yaml we have for example

GET /v1/things
POST /v1/things  //thingId as a parameter of the request body with the name id 
GET /v1/things/{thingId}

GET /v1/stuff/things
POST /v1/stuff/things //thingId as a parameter of the request body with the name id 
GET /v1/stuff/{thingId}

And the annotation file:

{
    "x-restler-global-annotations": [
      {
        "producer_endpoint": "/v1/things",
        "producer_method": "POST",
        "producer_resource_name": "/id",
        "consumer_param": "thingId"
      },
      {
        "producer_endpoint": "/v1/stuff/things",
        "producer_method": "POST",
        "producer_resource_name": "/id",
        "consumer_param": "thingId"
      }
    ]
  }

This kind of structure will result in a grammar that references the thingId of the /v1/things with the `GET /v1/stuff/things/{thingId} which of course will return a 404.

I resolved this temporarily renaming the parameter for /v1/stuff/things from thingId to stuffThingId. Still I wanted to ask if there is a possibility to let RESTler also look at the path of the request to reference the producer, and if a producer is not existent in a path (No post request for example) that the fallback reference is just done by name.

But maybe this is kinda to complicated and unnecessary. :)

[samuelspagl]

I think my last comment ist unnecessary.

Everything works as expected, thus I think this issue can be closed.

Thanks for everything!