Test definitions - implementations guidelines and artifacts

[rartych]

Problem description Delivery of API test cases and documentation is one of criteria in API Readiness Checklist. Each subproject needs to develop a Gherkin feature file for specified API. If subprojects also intend to add test implementations, an aligned single implementation that is agreed amongst all provider implementors could be added to the main subproject repo.

Possible evolution Definition of common guidelines and artifacts that would simplify development of test definition, documentation and implementation based on current experience in subprojects and within participating organizations.

Additional context As listed in https://github.com/camaraproject/SimSwap/issues/63 following projects already provide Gherkin feature files:

• EdgeCloud (Traffic Influence)
  • QualityOnDemand
  • HomeDevicesQoD

[bigludo7]

Hello @rartych and team, How we can move forward on this? as I assume this is the only missing point in a lot of APIs to reach completion.

Looking in details of the Test Case already contributed and seems we are not far from one WG to another - need few alignements.

• There is a lot similarities between https://github.com/camaraproject/HomeDevicesQoD/blob/main/code/Test_definitions/home_devices_qod_setQos.feature , https://github.com/camaraproject/SimSwap/pull/68 (provided by GSMA) and https://github.com/camaraproject/DeviceLocation/pull/119. The difference are very little (use of first person or third person, payload provided or described).
• The Test Case file in QoD (https://github.com/camaraproject/QualityOnDemand/blob/main/code/cucumber/src/test/resources/feature/QoD_API_Test.feature) is slightly different as it's a bit light on the 'when' description and for one scenario can have several when/then -The one for traffic influence is very detailed (same grammar) and also feature several when/then for a scenario - https://github.com/camaraproject/EdgeCloud/blob/main/code/Test_definitions/Traffic_Influence_Test.feature
• The file provided by Telefonica in SimSwap (https://github.com/camaraproject/SimSwap/pull/70) is close but use a specific clause [context] that probably we should avoid?

We need to agree on:

• Having one feature file per API or Resource ? or we keep this flexible and it's up to each project to manage this? (I prefer this later)
• probably good for consistency to use all same pronoun (I or impersonal)
• Scenario identifier. To trigger the discussion I propose '@'<(mandatory)name of the resource><(mandatory)number XX><(optional)short detail> (like @check_simswap_01_Verify_Swap_True_Default_MaxAge)
• Do we agree to have several when/then in one scenario ?
• Providing either API literal request value (with exemple) or payload 'ready to use' (as in home device QoD) is fine?
• ...

Definitely looking for consumer of these file to get their perspective. The less they have to do to use these files the better it will be.

[mdomale]

@bigludo7 Having one feature file per API or Resource ? or we keep this flexible and it's up to each project to manage this? (I prefer this later)

• Having multiple feature inside single feature file is unattainable because if we generate various characteristics within a single feature document, an gherkin parser error will occur when executing cucumber scenarios . Therefore, the response is negative. One feature file per API is advisable so that all scenarios can be covered correspoding to the API & corresponding resource.(Reference blog https://copyprogramming.com/howto/multiple-feature-inside-single-feature-file#multiple-feature-inside-single-feature-file )

probably good for consistency to use all same pronoun (I or impersonal)

• Third Person pronoun is advisable although there is no restriction to use First(I)person pronoun(Originator of BDD suggests First person pronoun usage). It is important to preserve consistency between the description of the scenario and its steps (do not switch grammatical persons), follow the criteria used if we are adding scenarios to an existing project, and favor clarity of what is written. (Ref-https://www.testquality.com/blog/tpost/v79acjttj1-cucumber-and-gherkin-language-best-pract)

Scenario identifier. To trigger the discussion I propose '@'<(mandatory)name of the resource> <(mandatory)number XX><(optional)short detail> (like @check_simswap_01_Verify_Swap_True_Default_MaxAge)

• As per https://support.smartbear.com/cucumberstudio/docs/tests/best-practices.html#scenario-content-set-up-writing-standards We can write tags in lower case and separate words with hyphens and as per above recommendation except optional text we can follow as it is because optional text(short detail) is already covered in scenario/scenario outline.

Do we agree to have several when/then in one scenario ?

• Cucumber official documentation at https://cucumber.io/docs/gherkin/reference/ recommends to use only one When due to the fact that only behavior is listed in one acceptance criteria. This is just a recommendation, but not a rule. And Cucumber documentation is mostly focused on the Acceptance criteria specification, but not the test automation. So, it is definitely possible to follow how it best suits your requirement when it comes to automation testing. I'd recommend using fewer "And" keywords by merging different steps. Below points I recommend (It is a recommendation, but not a rule :) ): Use only one Given, When and Then keywords in one flow of a scenario, use "And" keyword if you need to specify extra steps at the appropriate event If you come across using too many "And" keywords, try to merge multiple such steps. On side note When there is only one When in the test automation, then there are slight chances of :
• Number of tests in the automation increased.
• Total execution time for the automation goes up.
• We may be doing a lot of redundant actions across multiple scenarios.

Providing either API literal request value (with example) or payload 'ready to use' (as in home device QoD) is fine?

• Our recommendation is to have API literal request value but in case of exceptions it is fine to use ready payload as well. @akoshunyadi @shilpa-padgaonkar @rartych

[bigludo7]

Thanks @mdomale for your comments and the url sources which are very helpful

To sump-up:

• Proposal is to manage single feature file (Having multiple feature inside single feature file is unattainable)
• Third Person pronoun is advisable as using the third person, particularly in English, conveys information in a more official and impartial manner
• about scenario identifier, following your recommendation it should be '@'<(mandatory)name of the resource> -<(mandatory)number XX>-<(optional)short detail in lower case and using - separator> (like @check_simswap-01-verify-swap-true-default-maxAge) - As we use '_' in resource using then "-" could be confusing - WDYT?
• I'm not sure we can enforce any rule about using only one Given, When and Then keywords in one flow of a scenario but got your point to provide this a a recommendation.

Expecting also comments from others companies involved in day to day CAMARA APIs crafting :)

[shilpa-padgaonkar]

Thanks @mdomale and @bigludo7 for your inputs and feedback. While we wait for few more days to get more feedback on the inputs, I would also propose to create a short Camara-API-Testing-Guidelines.md doc with the below content:

• Short intro about the API test cases requirements specified in https://github.com/camaraproject/Commonalities/blob/main/documentation/API-Readiness-Checklist.md
• Location of the feature file in our subproject repos
• Best practices and recommendations section, which will incorporate all the above specified inputs

With this doc in place, we would not need to search for old issues regarding these discussed points when needed later, and would also allow the new Camara members to get an overview on the topic.

[jlurien]

Answering also to @bigludo7's questions. I think we are quite aligned with previous responses:

• Having one feature file per API or Resource ? or we keep this flexible and it's up to each project to manage this? (I prefer this later)

We also prefer one feature file per operationId.

• probably good for consistency to use all same pronoun (I or impersonal)

Yes, we should agree on a common wording. The simpler the better.

As a example, to be refined:

Background: (assuming one feature file per operationId)
  Given:
    the environment 
    the path "path"
    the method "method"

Given:
  the (path param|query param|header) "name" is set to "value"
  the request body is set to
    """
    """
When:
  a "operationId" request is sent

Then:
  the status code is "X"
  the response value at "JSON-path" is "value"

• Scenario identifier. To trigger the discussion I propose '@'<(mandatory)name of the resource>< (mandatory)number XX><(optional)short detail> (like @check_simswap_01_Verify_Swap_True_Default_MaxAge)

We think it's good to include the operationId of the endpoint, along with some number to allow unique identification, and some description may also help, so a proposal could be:

@.[.] (separator here could be other than "." but we should choose one that cannot be part of the components)

• Do we agree to have several when/then in one scenario ?

It is preferable to have simple scenarios with just 1 when. While testing APIs, the structure is usually:

Given: request setup When: request is sent Then: response is validated

If some complex scenario requires that filling a request depends on the response from a previous one, we may allow several when

• Providing either API literal request value (with exemple) or payload 'ready to use' (as in home device QoD) is fine?

For the Given steps, in some cases we may give exact values for some parameters, for example providing wrong values to provoke an error, while in other cases the exact value cannot be known in advance and we have to ask the tester to fill the value. For example, in location-verification we should ask the tester to provide the well known latitude, longitude of a device to test a verificationResult=true scenario.