postmanlabs / newman

Newman is a command-line collection runner for Postman
https://www.postman.com
Apache License 2.0
6.88k stars 1.17k forks source link

Pre-request and test scripts still count towards summary totals after being removed. #3004

Open jordanwalsh23 opened 2 years ago

jordanwalsh23 commented 2 years ago

When pre-request and test scripts are removed from Postman, newman still counts these towards the summary totals in the CLI reporter.

  1. Newman Version (can be found via newman -v): v5.3.0
  2. OS details (type, version, and architecture): MacOS 11.6.8 (Big Sur) Intel
  3. Are you using Newman as a library, or via the CLI? CLI
  4. Did you encounter this recently, or has this bug always been there: Recently
  5. Expected behaviour: Any scripts removed from Postman should not count towards the summary totals in the CLI reporter.
  6. Command / script used to run Newman: newman run
  7. Sample collection, and auxiliary files (minus the sensitive details): Any collection where the tests or pre-requests are entered, saved and then deleted.

Steps to reproduce the problem:

  1. Create a collection with a single request in postman (no tests yet)
  2. Export the collection
  3. Run in newman - note the counts in the CLI reporter show '0' for both tests and pre-request scripts (image 1)
  4. Update the request to have content in the pre-request and test scripts (e.g console.log).
  5. Export the collection
  6. Run in newman - note that the counts for the CLI reporter show '1' for both tests and pre-request scripts (image 2)
  7. Update the request to remove the content from pre-request and test scripts so they are empty.
  8. Export the collection
  9. Run in newman - note that the counts for the CLI reporter still shows '1' for both tests and pre-request scripts (image 3)

image image image

jordanwalsh23 commented 2 years ago

I believe this is because after the pre-request and test scripts are removed via Postman, the collection JSON still contains the following:

"event": [{
    "listen": "prerequest",
    "script": {
        "exec": [
            ""
        ],
        "type": "text/javascript"
    }
},
{
    "listen": "test",
    "script": {
        "exec": [
            ""
        ],
        "type": "text/javascript"
    }
}],

I believe the exec arrays are still being counted even though there is no executable javascript within these blocks.

If the event section is removed, the counts appear as expected.