search

systelab / allure-reporter

Printable report generation for Allure
5 stars 2 forks source link

Re-define the use of 'Actual Results' information #46

Closed ManelMoreno closed 4 years ago

ManelMoreno commented 5 years ago

It is needed to re-define what information do we want to be uploaded with Test Runs to 'Actual Results' field in JAMA.

  1. Review current info being uploaded.
  2. Provide ways to include Location of Test Evidences (logs, allure report containing input&output data, db, screenshots, etc.)
  3. Provide ways to include Test Environment used

This is for each Test Run

ManelMoreno commented 5 years ago

After the meeting with Olga, Dani, Ainhoa and myself, these are the actions derived (please modify with your comments/considerations):

  1. Remove the current content of 'Actual Results' (stop setting the table with the class/function name and Action)

  2. Automated Test Run must have information related to:

    • Test Environment used to run the Test Case. Could be:

      • a static text indicating that the detailed in JAMA test plan have been followed,
      • or that is detailed in the same Allure Report (indicated where it is)
      • explicit detail of Envir.used (HW/SW/SN/Configuration/DBs/Tools/etc..)

    • Indication where Evidences are located (mandatory for Medical Devices, optional/wished for the rest). Could be a sentence like: "See Test Execution evidences in: \sltrep...xxx". Allure report is a mandatory evidence of automation execution and must contain the specific values used for testing.

    • To satisfy the requirement of filling the result for each step, Actual Results could also contain the text: "Executed all steps with results as expected", for PASS status.

The technical solutions proposed are:

  1. Include a new text field in the UI of Allure reporter, in order to allow writing directly the desired text that will contain 'Actual Results' field for each of the Test Runs we are going to upload.
  2. Use/implement more labels coming in the json, containing specific details about Test Environment (currently, date, Tester, Browser, etc. are already implemented).
  3. Test automation implementation will feed text or parameterized text that will contain 'Actual Results' field for each of the Test Runs we are going to upload.
    • This one shall be compatible with option 1, to append and not overwrite both contents.
olgapuig commented 4 years ago

Potential solution: • Add the new field “Actual Results” as Text Area in the form to upload results to Jama • Add a new label “Actual Results” to the JSON/XML that automatically updates the Actual Results field in the Test Run in Jama. This information will be automatically added to the Test Run in Allure. The Test Design does not include this information. • When the “Actual Results” is informed in the JSON/XML, the information entered into the “Actual Results” in the UI must be concatenated.

ManelMoreno commented 4 years ago

We(Olga, Susana &Manel) met on 11/12/2019 and decided the solution to be implemented (see details above) mainly based in the proposals. The content of field 'Actual Results in JAMA will be filled by either a label in JSON/XML and/or A new free text in the Allure Report UI. No more input sources. Current table will not more appear. All the labels informed in the code (and so in JSON) will appear in Allure report for each Test Run. This should be enough to allow any project to report without manual effort all the required information to formalize automated TCs in Jama.

About Executed in version, we decide a new way to report ir. See more info in a different issue ().

olgapuig commented 4 years ago

Final solution:

ManelMoreno commented 4 years ago

Great Olga! Do we have it deployed in order to make some test?

olgapuig commented 4 years ago

Yes, it’s automatically deployed at the test environment when a Pull-request is performed. Refer at https://systelab.github.io/allure-reporter-test/. There’s still a minor change to develop, which is to have a proper display in Jama when it’s a multi-line text considering the line-breaks.

ManelMoreno commented 4 years ago

This implementation is to be verified. Please add here the cases tested and the results.

ManelMoreno commented 4 years ago

Verification done, no problems found except cosmetics (one of them fixed in an internal cycle. Verified several cases described in the attached file using XMLs and JSON, reporting Actual Results texts from the files and/or from UI (new field added into Allure Reporter UI form) Verification AR Results.txt

ManelMoreno commented 4 years ago

@SCoboG : please review the verification done and move it to Close status.

ManelMoreno commented 4 years ago

This feature has already been used by other test members in Synapse and QL4 projects, so I can consider it validated.