search

petarmaric / smoke_test

Console app and Python API for automated smoke testing
BSD 3-Clause "New" or "Revised" License
2 stars 2 forks source link

Adding data fixture(s) to test the number of arguments #3

Open RadeKornjaca opened 4 years ago

RadeKornjaca commented 4 years ago

Currently, the number of arguments that are provided in the example code match the "happy path" scenario. Using already provided mechanisms, add data fixture that will:

  1. Test if program returns specified error code if there are more arguments than required
  2. Test if program returns specified error code if there are less arguments than required

Example for the code from example directory (not necessarily 100% correct):

less-cmd-arguments-provided:
  args: $input_filename
  expect-returncode: 11

In case of accepting the #2, add the according stderr output to the fixture.

petarmaric commented 4 years ago

I fully agree that student CLI argument parsing ought to be tested as well. To be honest, I'm not sure why I didn't consider it myself (probably ran out of coffee).

Some thoughts/questions:

  1. I think a single test class should be able to cover both of your use cases, when less/more CLI arguments are given than needed. We can differentiate between these use cases simply by varing the expected-returncode test fixture data.
  2. Which class, within the smoke_test class hierarchy, would you suggest we subclass for these tests? My gut feeling is that we don't really need this feature within the smoke_test.tests.functional.stdio module.
  3. When creating the demo YAML test fixture for this I propose we have test cases for [0..N-1, N+1, N+2] CLI arguments, where N is the correct number of CLI arguments.
  4. Consider leaving stdout/stderr out of this issue, if you look at https://github.com/petarmaric/smoke_test/blob/1.0.1/smoke_test/tests/functional/file.py#L59 you'll see that the BaseFileErrorHandlingTest class and its descendants (FileErrorInputNotReadableTest and FileErrorOutputNotWritableTest) ignore stdout output and focus on expected-returncode exclusively. This is intentional, as from my experience you can't rely upon students to honor error message formatting.