search

apideck-libraries / portman

Port OpenAPI Specs to Postman Collections, inject test suite and run via Newman 👨🏽‍🚀
http://getportman.com/
Apache License 2.0
638 stars 59 forks source link

Any suggestions how to generate tests for invalid and blank values in request payload in fuzzy tests #564

Open ehlmnq opened 8 months ago

ehlmnq commented 8 months ago

Hi, Team

I have certain scenarios in which user wants to validate the invalid and blank values in request payload and then check the responses for correct messages for example :

  1. {"identifierType":"LOGINID","identifierValue":"XXXXX7771316118","workspace":"Inavalid"}
  2. {"identifierType":"LOGINID","identifierValue":"XXXXX7771316118","workspace":"ADMIN","langauge":""}

Is it possible to generate tests to validate using Fuzzy tests, and if possible please suggest some help.

thim81 commented 8 months ago

Hi @ehlmnq

The fuzzy generation is based on the OpenAPI definition. If the property has minLength option defined in OpenAPI, it will generate a fuzzy variance with a less characters. If the property is marked as required, Fuzzing will drop the property.

For "blank", we don't have any support yet, but we have received this request already a couple of times.

how would you want configure this? Would you add a certain x-portman-fuzz-blank: true to your OpenAPI or configure it in the Portman config, where you define for which properties?

The insertion "invalid" values is more complex, since what would be the definition of "invalid"? Would it be for ENUM values?

type: object
properties:
  status:
    type: string
    enum: ['Pending', 'Processing', 'Completed', 'Failed']

How would you like to configure it? Would it be based on OpenAPI

thim81 commented 8 months ago

Linked to https://github.com/apideck-libraries/portman/issues/555