Feature request: Code generator for Postman Sandbox API

[losinggeneration]

It would be useful to be able to turn a Postman request into something that could be used in Pre-Request or Test scripts. This can currently be done by selecting Code->NodeJS->Request, but the generated code needs changes to make it work as a Pre-Request or Test script. One example of this is that environment variables used in the original Request are expanded. The NodeJS Request options object also aren't exactly the same as the Sandbox pm.sendRequest's options object.

NodeJS Request

var request = require("request");

var options = { method: 'DELETE',
  url: 'http://local.dev/all_the_things',
  headers: 
   { 'postman-token': '75c3858e-24b9-4208-86f3-3ec98f42bc27',
     'cache-control': 'no-cache',
     'env_header': 'SomeEnvironmentHeader',
     'authorization': 'bearer SomeTokenStoredInTheEnvironment',
     'content-type': 'application/json' } };

request(options, function (error, response, body) {
  if (error) throw new Error(error);

  console.log(body);
});

Postman Request

var options = { method: 'DELETE',
  url: pm.environment.get("local")+'/all_the_things',
  header:
   { 'postman-token': '75c3858e-24b9-4208-86f3-3ec98f42bc27',
     'cache-control': 'no-cache',
     'env_header':  pm.environment.get("local_env_header"),
     'authorization': `bearer ${pm.environment.get("local_auth_token")}`,
     'content-type': 'application/json' } };

pm.sendRequest(options, function (error, response) {
  if (error) { console.log(error); return; }

  console.log(response.text());
});

There's a few things going on in this. It's using environment variables from the original request and using it in various ways (direct call only, concatenation, & ES template.) Concatenation could be used, but I find it slightly less readable than using templates.

---

Postman for Linux Version 5.3.2 linux 4.13.0-1-amd64 / x64

[kamalaknn]

@losinggeneration Thanks for the suggestion. That would be an interesting use of code generation in Postman.

One more thought, can this request part of your collection. If we gave you more control over the execution order of a collection would that help? We are building something over postman.setNextRequest. We'll try to include this while building that.

[losinggeneration]

I think it could, yes. It may require us to structure our collections differently though. The main drawback, from our team's perspective with using multiple requests in a collection is sharing state between requests. The environment can be used, but it persists outside of the scope of those request(s).

I'll describe our main usage of Postman below so you have a better idea what we're doing, trying to accomplish, and some of the pros & cons we've found from our perspective.

User experience report

Usage

We're mainly using Postman for API system integration testing of one or more services. We have our stack spun up and let Postman or Newman run the collection(s) across our services. We often need add functionality to test to a folder that will include setup and teardown. In the case of testing a GET request, setup the data is done by first calling a POST (or sometimes multiple POST's.) Then the GET request(s) are called with tests on the request to verify the response & data. After all tests are complete, we often go into a cleanup/teardown request to delete created resources used for the tests. We then have other folders with distinct test(s) to test other endpoints with a similar setup/test/teardown structure. I imagine you've seen this type of flow before, so I'd assume it's not too surprising to hear.

We are often writing/updating a collection as new endpoints and often sets of endpoints are created. One example of a set of endpoints would be CRUD for a new resource. As we're implementing new endpoints that are part of a set, they are often in a self contained folder with dependencies being done as part of the setup to test each new endpoint. We try to make time to restructure these later on to remove some of the duplication that arrives from this. If this doesn't happen right away, it's often not a huge deal since each set are in its own sub-folder and each endpoint test is in a sub-folder of that.

Implementations of usage

Everything is a new request in the collection

We have two methods of accomplishing what I describe above. The first is to have each request in a folder make exactly one request. So, setup may be (using a fictional issue tracker as an example): Create Issue, Create Comment, and Add Attachment as a second comment. The tests would then verify that we can get all comments for the issue, test that we can get the first one by using limit & offset, and test that we can get the second one by using limit & offset (we may additionally test error cases of getting comments.) After that we have a request to delete the issue (assuming it cascades, otherwise it would be delete attachment, delete comment, delete issue.)

Pros

• Each request is easily visible
• Visibility makes it easier to find & duplicate requests to other parts of the collection that may need it
• Once each setup request is run, multiple integration requests can be made before finally calling teardown request(s)
• Easier to write & test setup requests
• Setup or teardown can often be easily derived from earlier tests. An example of this would be a test of a POST for creation is later used as a setup step to verify a GET works as intended. Sometimes this also allows us to verify CRUD endpoints by doing something like test Create, test Read for previous create (further validating the Create actually persisted), test Update, test Delete. We often verify that, for instance, Create, Update, & Delete really did the correct action to the resource by doing a Read. While this extra Read could be another request in the collection, to us, it makes more sense to have it in the Tests since seems to logically fit there as a "did this actually do the thing you said you did." We've been doing this with sendRequest because the Read doesn't need to duplicate tests for each one for verification (and there's less state to pass between requests.) This is one of the reasons why this feature request came about: turn a previously written Read request into a sendRequest to verify Create, Update, & Delete all are working as expected.

Cons

• Rerunning a single request may require rerunning setup in some cases. Similar to this, it's sometimes easier to rerun the folder from the collection runner, but IMO that interface makes it harder to understand why a request failed.
• State from setup often requires setting environment variables which may or may not get cleaned up during teardown. If they don't, they may linger well after the requests have been run. Mistakes when writing tests can then happen because a new request either forgets to set or typos an environment variable and uses an existing environment variable set elsewhere.
• There is often more coupling between requests that isn't always obvious to be able to rerun specific requests.

All contained in a single request

The other method we've done is the other extreme. We do all setup & teardown in each request. So the Pre-Request Script of each Request would include the pm.sendRequest's for creating the issue, comment, & attachment. The Tests would include a section to teardown the Request's data at the end. This is mostly useful for having setup and isolated testing of a specific endpoint, for instance, testing multiple paths from a GET resource.

Pros

• Requests are self contained to allow them to easily be run from both the runner & the main interface.
• No state shared between requests.

Cons

• Hinders visibility of setup/teardown
• Similar to above, lack of visibility in setup/teardown makes finding requests that are reusable more difficult.
• If pre-request setup or post request teardown fails, it's harder to track down why.
• Significantly increases collection run time.
• sendRequest, IIRC, is asynchronous, which means we have to be a bit careful with how multiple calls are structured.
• Each request is often unique and hard to reuse later without changes to the Pre-request/Tests scripts.

Hybrid

One thing we've talked about, but haven't really done is a mix of the two. A single setup request & a single teardown request with all endpoints called within either the Pre-Request Script or Tests to make the multiple pm.sendRequest calls. Then multiple calls can be made without having to have each request making potentially many sendRequest calls. This is mostly useful for having setup and isolated testing of a specific endpoint, for instance, testing multiple paths from a GET resource.

Pros

• Setup is a single (and teardown another) Request in a collection making setup a bit easier from the main interface.
• State between calls is mostly created via the single Setup Request.

Cons

• Hinders visibility of setup/teardown
• Similar to above, lack of visibility in setup/teardown makes finding requests that are reusable more difficult.
• If setup/teardown fail, it's harder to track down why.
• sendRequest, IIRC, is asynchronous, which means we have to be a bit careful with how multiple calls are structured.

Conclusions

Using Postman for integration testing has worked pretty well for us. We've had a few issues here-and-there (mostly with how to organize tests.) As you can see from above, we don't really have a single organization method for verifying an API is working as intended. However, through this mix of organizations we've been able to have pretty good success with using Postman as a system integration tester. Having the ability to generate a sendRequest from a request in a collection seems like it would help us with our current collection structure.

[kamalaknn]

@losinggeneration Thanks for your detailed report. This gives great insights on how Postman collections are structured and used.

I'm forwarding this @a85 and @shamasis. We are building a set of features for reusability and organization in Postman Collections, we'll include this in our design discussions.

[mccannt]

+1