googleapis / sample-tester

Tool for testing semantically equivalent samples in multiple languages and environments
Apache License 2.0
10 stars 12 forks source link

Allow defining custom YAML directives #65

Open vchudnov-g opened 5 years ago

vchudnov-g commented 5 years ago

Allow the sample test file to include sample YAML directives that can modify the test state. For example:

- define: ensure_foo_is_set_up
  - shell:
    ...
  - call:
    ...
  - code: |
    # do something
    foo = 'whatever'  # modifies the workspace for later directives

This will also benefit from having a testplan file able to include other files which, for example, define helper directives. This could done via any of:

  1. an include directive that inserts files (provided as a list) at a point (like #include in C) or
  2. maybe actually using cpp and #include in the yaml or
  3. a top-level definitions key that lists files with - defines to process before parsing the test suites/cases
vchudnov-g commented 5 years ago

A good plan of action might be to implement - define first and then use (2) (#include + cpp) for now to see how it works with template files. Then we can decide about moving to (1) or (3)

vchudnov-g commented 5 years ago

@beccasaurus @googleapis/samplegen

beccasaurus commented 5 years ago

Yaaaaassssss

beccasaurus commented 5 years ago

I wouldn't actually use cpp and #include :)

# sample_with_defined_methods.yaml
test:
  suites:
  - name: Dog Tests

    helpers:
      create_dog_helper:
      # is this added to the overall scope? or isn't it? there are use-cases for both cases! hmm. no?
      - uuid: dog_identifier
      - # ...
      - call:
           sample: the_create_dog_sample # or shell something or whatever
           params:
             # this can access a scope of method locals (from the parameter names + whatever gets
             # defined in the scope of running this, like a uuid/env/etc) + the running scope
             # (e.g. of the test case or setup being executed) with the locals > the global.
             # regular variable scope behavior! maybe :P
             dog_name: { variable: name } # variable comes from local method/definition/whatever scope
             dog_breed: { variable: breed }
             dog_identifier: { dog_identifier }

    setup:
    - create_dog_helper:
        params:
          name: { literal: Rover }
          breed: { literal: Golden Retriever }

     cases:
     - name: List Dogs
       spec:
         call:
           sample: list_dogs
         assert_contains:
         - literal: Rover

# sample_which_imports_methods.yaml
test:
  suites:
  - name: Dog Tests

    imports:
    - ../shared/managing_dog_helpers

    setup:
    - create_dog_helper:
        params:
          name: { literal: Rover }
          breed: { literal: Golden Retriever }

     cases:
     - name: List Dogs
       spec:
         call:
           sample: list_dogs
         assert_contains:
         - literal: Rover
# ../shared/managing_dogs_helpers.yaml
test:
  helpers:
    create_dog_helper:
    # is this added to the overall scope? or isn't it? there are use-cases for both cases! hmm. no?
    - uuid: dog_identifier
    - # ...
    - call:
         sample: the_create_dog_sample # or shell something or whatever
         params:
           # this can access a scope of method locals (from the parameter names + whatever gets
           # defined in the scope of running this, like a uuid/env/etc) + the running scope
           # (e.g. of the test case or setup being executed) with the locals > the global.
           # regular variable scope behavior! maybe :P
           dog_name: { variable: name } # variable comes from local method/definition/whatever scope
           dog_breed: { variable: breed }
             dog_identifier: { dog_identifier }

Braindumping thoughts while they're on my mind.

The term "helpers" could be something else. My mind defaulted to "helpers" because it's pretty common terminology for methods you might add to your test suite for doing this sort of thing.