search

postmanlabs / newman

Newman is a command-line collection runner for Postman
https://www.postman.com
Apache License 2.0
6.82k stars 1.15k forks source link

JSON loaded from external file doesn't unwrap when passed to programmatic runner #3179

Open jfo opened 9 months ago

jfo commented 9 months ago

When a collection is contained in an external JSON file, the behavior wrt how it is processed differs between the CLI interface and the programmatic interface in a surprising way.

When passed as a filepath, the collection can be loaded if it's in a JSON object OR if it's in a JSON object under the key collection, but when passed as JSON directly, this unwrapping doesn't happen. The unwrapping happens here:

https://github.com/postmanlabs/newman/blob/a612fbe29348eaae5f6aac0b882382d23d688ac3/lib/run/options.js#L60

This leads to surprising behavior, as my expectation was that running a json file that works on the CLI would also work when passed as part of a script. There is no feedback that the formatting of the json could be the issue and be causing the difference.

  1. Newman Version (can be found via newman -v): 6.0.0

  2. OS details (type, version, and architecture): Mac Ventura M2

  3. Are you using Newman as a library, or via the CLI? Both, see issue

  4. Did you encounter this recently, or has this bug always been there: Idk

  5. Expected behaviour: Consistency between CLI and programmatic use

  6. Command / script used to run Newman:

  7. Sample collection, and auxiliary files (minus the sensitive details): reproduction

Reproduction can be found in this diff in this branch

Wrapped JSON:  {
  iterations: { total: 0, pending: 0, failed: 0 },
  items: { total: 0, pending: 0, failed: 0 },
  scripts: { total: 0, pending: 0, failed: 0 },
  prerequests: { total: 0, pending: 0, failed: 0 },
  requests: { total: 0, pending: 0, failed: 0 },
  tests: { total: 0, pending: 0, failed: 0 },
  assertions: { total: 0, pending: 0, failed: 0 },
  testScripts: { total: 0, pending: 0, failed: 0 },
  prerequestScripts: { total: 0, pending: 0, failed: 0 }
}
Bare JSON:  {
  iterations: { total: 1, pending: 0, failed: 0 },
  items: { total: 3, pending: 0, failed: 0 },
  scripts: { total: 1, pending: 0, failed: 0 },
  prerequests: { total: 3, pending: 0, failed: 0 },
  requests: { total: 3, pending: 0, failed: 0 },
  tests: { total: 3, pending: 0, failed: 0 },
  assertions: { total: 2, pending: 0, failed: 0 },
  testScripts: { total: 1, pending: 0, failed: 0 },
  prerequestScripts: { total: 0, pending: 0, failed: 0 }
}
Wrapped: Filepath {
  iterations: { total: 1, pending: 0, failed: 0 },
  items: { total: 3, pending: 0, failed: 0 },
  scripts: { total: 1, pending: 0, failed: 0 },
  prerequests: { total: 3, pending: 0, failed: 0 },
  requests: { total: 3, pending: 0, failed: 0 },
  tests: { total: 3, pending: 0, failed: 0 },
  assertions: { total: 2, pending: 0, failed: 0 },
  testScripts: { total: 1, pending: 0, failed: 0 },
  prerequestScripts: { total: 0, pending: 0, failed: 0 }
}
Bare Filepath:  {
  iterations: { total: 1, pending: 0, failed: 0 },
  items: { total: 3, pending: 0, failed: 0 },
  scripts: { total: 1, pending: 0, failed: 0 },
  prerequests: { total: 3, pending: 0, failed: 0 },
  requests: { total: 3, pending: 0, failed: 0 },
  tests: { total: 3, pending: 0, failed: 0 },
  assertions: { total: 2, pending: 0, failed: 0 },
  testScripts: { total: 1, pending: 0, failed: 0 },
  prerequestScripts: { total: 0, pending: 0, failed: 0 }
}

I don't have the context to know what the expected behavior should be, other than I think passing a path and passing a JSON object should work consistently. I can see there is a lot of backwards compatibility stuff going on too. I'm filing this issue so that there is some visibility into the surprising behavior.

Thanks!