Xray integration - Githubissues

CMCunha commented 2 years ago

I have one suggestion to have a better synergy between Xray and Karate for you to consider.

In order to have a better experience between the tools it could be useful for the users to be able to refer a test case or requirement in Xray to connect to the Karate test, with this change users that want to link the Karate test case to a test case in the Xray side can by using tags, in the same way if they want to link a test to a requirement they could also using tags. I added a possible example of this for you to consider, in the feature file we can see new tags: @xray_requirement=CALC-1 @xray_test=CALC-2

Feature: sample karate test script

Background:

* url 'http://dummy.restapiexample.com/api/v1/'

@xray_requirement=CALC-1 @xray_test=CALC-2
Scenario: get all dummy users and then get the first user by id
Given path 'employees'
When method get
Then status 200

* def first = response.data[0]

Given path 'employee', first.id
When method get
Then status 200

That, when those tags exist, must be added to the report like we can see below:

<testsuite failures="0" name="examples/DummyUsers/dummyusers.feature" skipped="0" tests="2" time="3.646495">
    <testcase classname="examples.DummyUsers.dummyusers" name="[1:7] get all dummy users and then get the first user by id" time="2.036197">
        <properties>
             <!-- using a custom "test_key" property -->
             <property name="test_key" value="CALC-2" />
            <!-- using a custom "requirement" property -->
             <property name="requirement" value="CALC-1" />
        </properties>
        <system-out>
        * url 'http://dummy.restapiexample.com/api/v1/' ........................... passed
        Given path 'employees' .................................................... passed
        When method get ........................................................... passed
        Then status 200 ........................................................... passed
        * def first = response.data[0] ............................................ passed
        Given path 'employee', first.id ........................................... passed
        When method get ........................................................... passed
        Then status 200 ........................................................... passed
        </system-out>
    </testcase>
(...)
</testsuite>

Once this information is added to the report it will be used in the Xray importation and link the test to the requirement and test.

ptrthomas commented 2 years ago

I'd like the community to comment - will adding these extra XML elements break any tools that depend on the JUnit XML

bitcoder commented 2 years ago

Just to add here some info and some thoughts. Properties at testcase level are not common even though some other tools (e.g. Datadog) also follow a similar approach. Tools making a strict validation of the schema may not like this but AFAIK it should not be a issue. More, my recommendation would be to only add the <properties> element under the <testcase>, if any of those properties is provided. That way, we would ensure the report stays as it is for users that don't use these new tags.

borzykin commented 2 years ago

Why add support for Xray particularly when it can be added in a more generic way for all TMS, like it is done in allure: https://docs.qameta.io/allure/#_links

bitcoder commented 2 years ago

Well @borzykin , it's true that a generic way could be applied but it all depends on the goal. If the goal was to provide a unidirectional link, so it would be present on the Karate report on the respective Scenarios to have a quick way to jump to the test management tool, that approach would perhaps make more sense.

However, here the goal is not just to provide that link on the report; the goal is to be able to report results to existing tests in Xray/Jira that someone decided that should be automated, so that the results will be added to them. More, also to automatically cover a existing Story/requirement right from the code in order to have to avoid doing it manually... so that whenever results are imported, the corresponding Test in Xray/Jira will be linked back to the identified story. These are scenarios are highly common among teams, based on our experience. The mentioned goal requires that the information that is passed to Xray (i.e. the JUnit XML report in this case) has some specific information that Xray can process in order to link the the results back to the correct entities.

mj11pyqe commented 2 years ago

I tried linking scenarios to xray. With Junit report, the tickets were identified by feeature pathname. Was not very refactoring friendly. Then I moved to cucumber report. With cucubmer, tickets were linked to tags correctly, but can tickets had to be predefined already. So I had to create a script that makes a xray json.

I don't know if Xray now can identify Junit report by tags. If yes, upvote for this suggestion. If now, you better of using cucumber report, as it have all the tags needed.

utktrl commented 2 years ago

I agree that this change will be very helpful. And I hope Xray will also enable this feature on Xray Server as well.

bltemrah commented 2 years ago

Hi @CMCunha and @ptrthomas Is there any way that we integrate our automation test results with Xray?

CMCunha commented 2 years ago

Hi @bltemrah, Please check this tutorial: https://docs.getxray.app/display/XRAYCLOUD/Testing+APIs+using+Karate+DSL that we did showcasing a possible integration (that as originated this thread actually).

bltemrah commented 2 years ago

@CMCunha Thank you for your prompt response. I found a valuable information in what you addressed. However, my FW based on gradle build which does not support XML report, I can only reach HTML report. Can you please let me know how I can integrate the gradle FW with Jira Xray? Thanks in advance

CMCunha commented 2 years ago

@bltemrah I'm glad you find valuable information in the documentation. We also have a tutorial that can help you to integrate Gradle with Xray: https://docs.getxray.app/display/XRAYCLOUD/Integration+with+Gradle.

bltemrah commented 2 years ago

@CMCunha thank you. We use Karate FW with Gradle. So my question is here which one do I need to consider Gradle? or Karate fw? I lost myself in the below two web pages :) If Gradle works well, why do we need to apply the below? https://docs.getxray.app/display/XRAYCLOUD/Testing+APIs+using+Karate+DSL If the below Karate fw approach/solution works well, why do we need to use Gradle? https://docs.getxray.app/display/XRAYCLOUD/Integration+with+Gradle. I just confuse which one is important the build way or fw itself. Thanks in advance

CMCunha commented 2 years ago

@bltemrah Xray is agnostic of the framework or tool you are using as long as it produces a compatible test report (junit, xunit, nunit, robot, behave, etc) that can be interpreted by Xray. So if you are using Karate DSL to perform the tests and generate a Junit report you can follow this tutorial https://docs.getxray.app/display/XRAYCLOUD/Testing+APIs+using+Karate+DSL to see how to upload those results to Xray using an API call. If you are using Gradle to compile and execute the tests and want to upload the results from the Gradle scripts you can do it following this tutorial https://docs.getxray.app/display/XRAYCLOUD/Integration+with+Gradle.

These are two use cases that showcase two possibilities to integrate with Xray, you should check what is best for your case and adapt to your situation. If you prefer you can use the API and import the results on the best place on your process making an API call: https://docs.getxray.app/display/XRAYCLOUD/Import+Execution+Results+-+REST.

bltemrah commented 2 years ago

Thank you for your statement @CMCunha Actually, we need to run more than one "test case" in the one test scenario outline that has more than one scenario. We use "tag" for each test case, likes;

Today, I applied what guided in the below https://docs.getxray.app/display/XRAYCLOUD/Integration+with+Gradle with the below properties

Everything went well, and the report was directly integrated with the test plan in Xray. However, there is only one execution ticket created;

As I mentioned above, I run many test cases under this test scenarios. My question is " is there any way to create the different test execution tickets based on "tag" defined on the scenario? Thanks again

bitcoder commented 2 years ago

Whenever importing test results using a JUnit XML file, each Junit element on the JUnit XML report is mapped to a Test in Xray. @bltemrah can you please clarify exactly the scenario outline you have and how those tags are applied? I got a bit confused with: Actually, we need to run more than one "test case" in the one test scenario outline that has more than one scenario.

Note: usually, whenever you import results from a report file, only 1 Test Execution issue will be created; it will contain the test results (i.e. test runs), one per each testcase on the report... which in this case may correspond to a Scenario on your .feature file.

bltemrah commented 2 years ago

Hi @bitcoder For example, the below scenario outline has more than one test case, and each test case has its own "Tag";

Let me share what I understood. I need to separate each test case with one test scenario, otherwise, it is not possible to see their test result separately in different test execution under the test plan or test set in Jira-Xray based on their "tag"

bltemrah commented 2 years ago

Hi @CMCunha, I achieved the integration by using Jira UI way;

However, API approached did not work;

First I used the below authentication script to get token

curl -H "Content-Type: application/json" -X POST --data '{ "client_id": "API_KEY","client_secret": "API_SECRET" }' https://xray.cloud.getxray.app/api/v2/authenticate

and it returns a valid token.

After that, I used the below Post request script with the token received by using above script;

curl -H "Content-Type: text/xml" -X POST -H "Authorization: Bearer $token" --data @"junit-new.xml" https://xray.cloud.getxray.app/api/v2/import/execution/junit?projectKey=XT&testPlanKey=XT-351

I got the below error;

Can you please help me to solve this problem. Thanks in advance

bitcoder commented 2 years ago

Hi @bitcoder For example, the below scenario outline has more than one test case, and each test case has its own "Tag"; Let me share what I understood. I need to separate each test case with one test scenario, otherwise, it is not possible to see their test result separately in different test execution under the test plan or test set in Jira-Xray based on their "tag"

Yes, each Scenario is indeed a test scenario (or should be).

bltemrah commented 2 years ago

@CMCunha When I checked the below;

https://docs.getxray.app/display/XRAYCLOUD/Exporting+Cucumber+Tests+-+REST+v2 It shows an example request;

curl -H "Content-Type: application/json" -X GET -H "Authorization: Bearer $token" "https://xray.cloud.getxray.app/api/v2/export/cucumber?keys=DEMO-6;DEMO-7"

As you see, this script is particularly for cucumber. Do you think we can use it without "cucumber" as mentioned below;

https://docs.getxray.app/display/XRAYCLOUD/Testing+APIs+using+Karate+DSL#tab-API

As mentioned in my previous message, It did not work and returned the below error;

CMCunha commented 2 years ago

@bltemrah The cucumber example you are referring to is to export scenarios written in Xray to your code, the first example you were talking about was about importing test results to Xray. Are you writing scenarios in Xray and want to export those?

Regarding the error of the request you are showing it seems that you need to enclose the URL with " in order to be considered with all parameters.
Replace the last part from: --data @"junit-new.xml" https://xray.cloud.getxray.app/api/v2/import/execution/junit?projectKey=CQA&testPlanKey=CQA-811 to --data @"junit-new.xml" "https://xray.cloud.getxray.app/api/v2/import/execution/junit?projectKey=CQA&testPlanKey=CQA-811"

bltemrah commented 2 years ago

Hi @CMCunha Thank you so much. It works once adding URL btw "". For your 1st point, I do not write the scenarios in Xray. I really thank you for your assistance.

bltemrah commented 2 years ago

If you do not mind, I have 2 more quick questions @CMCunha

In the karate fw, we should run our tests only with/in ParallelRunner to generate .xml report in the "Build" folder.

Otherwise, the running test in the feature generates the test report under the "Target" folder with HTML type, NOT .xml format. Also, the test report is not created under "build" folder if we run the test in the feature;

As seen, we need .xml test report for the above script (as mentioned on the below page) to integrate and directly send our test report to Jira-Xray.

https://docs.getxray.app/display/XRAYCLOUD/Testing+APIs+using+Karate+DSL#tab-API

My question;

Is there any way to get .xml report of test results without using ParallelRunner? meaning directly run the test in the feature;

Can we use the scripts (as we have discussed in the previous msgs) with different test reports such as Json or Html? If yes, how can we get JSON report in karate fw with Gradle build? Under the "Target" folder, I can only have below;

Thanks again.

CMCunha commented 2 years ago

@bltemrah 1. Is there any way to get .xml report of test results without using ParallelRunner? meaning directly run the test in the feature; -> From my understanding you must use ParallelRunner to generate xml reports.

2. Can we use the scripts (as we have discussed in the previous msgs) with different test reports such as Json or Html? If yes, how can we get JSON report in karate fw with Gradle build? Under the "Target" folder, I can only have below; -> Xray does not support HTML or other custom reports, the full list of supported formats is here: https://docs.getxray.app/display/XRAYCLOUD/Import+Execution+Results+-+REST , so in order to upload results to Xray you must obtain one report that is compatible.

Make sure that you are using the .outputJunitXml(true) method to generate the xml report as it is described in the tutorial: https://docs.getxray.app/display/XRAYCLOUD/Testing+APIs+using+Karate+DSL (look at the class ExamplesTest.java).

bltemrah commented 2 years ago

Thank you, @CMCunha For the below URL, is it possible to Post a different ticket type than "testPlanKey", i.e. Test Set or Story?

"https://xray.cloud.getxray.app/api/v2/import/execution/junit?projectKey=CQA&testPlanKey=CQA-811"

CMCunha commented 2 years ago

@bltemrah No, it is not possible to define a Test Set or Story in the API call to Xray. Please check the possible parameters to use in the import request here: https://docs.getxray.app/display/XRAYCLOUD/Import+Execution+Results+-+REST#ImportExecutionResultsREST-JUnitXMLresults.

bltemrah commented 2 years ago

Hi @CMCunha, After running the below script to send our test scenario results to Jira Xray, it will create a test case for each test scenario.

curl -H "Content-Type: text/xml" -X POST -H "Authorization: Bearer TOKEN" --data @"io.token.features.adapter.obie.unitedKingdom.Natwest_PIS.xml" "https://xray.cloud.getxray.app/api/v2/import/execution/junit?projectKey=CQA&testExecKey=CQA-1077"

We do not expect that Jira Xray create a test case ticket for the same scenario. However, if its classname or name in the .XML report as seen below is different, Jira Xray creates a new test case again and again;

* def theBank = bankDetails.UnitedKingdom.Natwest ......................... passed * call read(makePayment) .................................................. failed Stack Trace: com.intuit.karate.KarateException: driver config / start failed: http call failed after 2 milliseconds for url: http://localhost:9222/json, options: {type=chrome, headless=false, target=null} classpath:io/token/webAppConsent/genericWebAppActions.feature:77 classpath:io/token/webAppConsent/unitedkingdom/NatWestBank_PIS.feature:51 classpath:io/token/features/generic/pisGeneric.feature:222 classpath:io/token/features/adapter/obie/unitedKingdom/Natwest_PIS.feature:9 at <feature>.: * call read(makePayment) (classpath:io/token/features/adapter/obie/unitedKingdom/Natwest_PIS.feature:9:9)

* def theBank = bankDetails.UnitedKingdom.Natwest ......................... passed * def enableOptionalFields = true ......................................... passed * call read(makePayment) .................................................. passed

* My question is how we ignore classname or name in the above script by adding something to prohibit a creating a different test case for the same test scenario? Thanks Emrah

bitcoder commented 2 years ago

Hi @bltemrah , Xray creates a Test entity based on the classname+name attributes of the <testcase> element on the JUnit XML report. If a Test already exists with that identifier (which is stored on the Generic Definition field on the Test issue), then it will just report results against it; otherwise, it creates/autoprovision the Test.

If your scenario results in <testcase> elements on the JUnit XML with different identifiers, then Xray will consider them as different tests; this is the standard behavior of Xray whenever handling JUnit XML files (more info here).

From my understanding, you wanted to have just one Test in Xray corresponding to the whole Scenario Outline that you have in karate, instead of having one Test per each row of the Example section of the scenario outline? Is that it?

I'm not sure exactly about the internals of karate and the current logic on how scenarios are mapped to testcase elements on the JUnit XML report.

bltemrah commented 2 years ago

Hi @bitcoder, First, thanks for your prompt response. Let me make it clear. We have the below test scenario (not scenario outline);

Scenario: PIS - sepa - TPP should successfully initiate with mandatory fields only

call read(makePayment)

I need to run this test scenario for several different banks such as HSBC, NatWest and Commerz. Even though I run the same test case, XML report has a different name and classname because of the changing of the Bank name in the below path for each Bank;

name="io/token/features/adapter/obie/unitedKingdom/Commerz_PIS.feature"

name="io/token/features/adapter/obie/unitedKingdom/HSBC_PIS.feature"

name="io/token/features/adapter/obie/unitedKingdom/NatWest_PIS.feature"

As you see, the test case is same, so it is better for us not to see/have more than one test case related to the same scenario even though we run it for the different banks. Can you please let me know how we can handle this issue? Is there any solution/way? Thanks in advance Emrah

bitcoder commented 2 years ago

@bltemrah , If possible, can you please shaer exactly your full Scenario specification ?

bltemrah commented 2 years ago

@bitcoder

Here it is;

Scenario: PIS - sepa - TPP should successfully initiate with mandatory fields only

call read(makePayment)

@makePayment And configure continueOnStepFailure = { enabled: true, continueAfter: true, keywords: ['match'] }

call read('classpath:io/token/features/generic/pisGeneric.feature@initiatePISRequest')
if (disableFutureDatedPaymentConversion == true) karate.call(true, "classpath:io/token/features/generic/pisGeneric.feature@futureDateConversionTrueScenario")
if (disableFutureDatedPaymentConversion == true) karate.abort()
call read(bankAuthToCall)
if (isBetaFdt) karate.abort()
if (isFdtTrue) karate.abort()
call read('classpath:io/token/features/generic/pisGeneric.feature@getTransferDetails') And match response.transfer.payload.amount.value == transferAmount And match response.transfer.providerDetails.status == '#notnull'
if (isReturnRefundAccount == true) karate.call(true, 'classpath:io/token/features/generic/pisGeneric.feature@checkReturnRefundAccount')
call read('classpath:io/token/features/generic/pisGeneric.feature@checkIfTransferStatusMatchProviderDetailsStatus')
if (makeFutureDatedPayment == true) karate.call(true, 'classpath:io/token/features/generic/pisGeneric.feature@checkFutureDatedPayment')
if (isCheckTransactionStatusSecondTime == true) karate.call(true, 'classpath:io/token/features/generic/pisGeneric.feature@checkTransferStatusA2ndTime')

@initiatePISRequest Scenario: Initiate PIS request

def isPisScenario = true
callonce read('classpath:io/token/features/generic/generic.feature@prepareBankAuthPathName')
call read('classpath:io/token/features/generic/pisGeneric.feature@defineBankData')
call read('classpath:io/token/features/generic/pisGeneric.feature@defineRequestPayload') Given url apiUrl+"/token-requests" And header Content-Type = "application/json" And header Authorization = apiKey
def apiKeyForConsent = apiKey
basePisRequestPayload.requestPayload.refId = 'R'+refId
basePisRequestPayload.requestPayload.description = 'D'+refId
basePisRequestPayload.requestPayload.to.alias.value = alias
basePisRequestPayload.requestPayload.to.id = memberId
basePisRequestPayload.requestPayload.redirectUrl = callbackUrl
basePisRequestPayload.requestOptions = bankId
call read('classpath:io/token/features/generic/generic.feature@defineAuthorizationPath') And request basePisRequestPayload When method post Then match responseStatus == 200
def tokenRequestId = response.tokenRequest.id

@futureDateConversionTrueScenario Scenario: Given futureDateConversion = true, this scenario to be called Given url apiUrl+"/token-requests/"+tokenRequestId+"/authorization" And header Authorization = apiKeyForConsent And header Content-Type = "application/json" And request { "consentAccepted":true } When method post Then status 400

match response == "FAILED_PRECONDITION: Bank " + bankIdPlain + outOfOurOperationalHouseMessage

@getTransferDetails Scenario: get token request result Given url apiUrl+"/token-requests/"+tokenRequestId+"/token_request_result" And header Authorization = apiKey

retry until responseStatus == 200 When method get Then match responseStatus == 200
if (karate.match("getBanksInfo.requiresOneStepPayment == '#notpresent'").pass == true) karate.call(true,'classpath:io/token/features/generic/pisGeneric.feature@redeemTransferTokenTwoStep')
def transferId = (karate.match("getBanksInfo.requiresOneStepPayment == '#notpresent'").pass == true) ? transferId : response.transferId Given url apiUrl+ "/transfers/"+ transferId And header Authorization = apiKey
retry until responseStatus == 200 When method get Then match responseStatus == 200

if you need other path, please let me know.

bitcoder commented 2 years ago

@bltemrah you shared 4 scenarios, therefore I would expect, generally speaking, one <testcase> entry on the JUnit XML report. Weren't you just referring to the scenario "PIS - sepa - TPP should successfully initiate with mandatory fields only" ? How do you run that scenario exactly against the different banks?

bltemrah commented 2 years ago

@bitcoder Actually, it is only one scenario which calls others. I just run the below scenario which includes one test case;

Scenario: PIS - sepa - TPP should successfully initiate with mandatory fields only

call read(makePayment)

Even though the name of scenarios is "PIS - Sepa - TPP should successfully initiate with mandatory fields only", the classname and name is seen differently;

<testsuite failures="3" name="io/token/features/adapter/obie/unitedKingdom/Natwest_PIS.feature" skipped="0" tests="9" time="317.715079"><testcase classname="io.token.features.adapter.obie.unitedKingdom.Natwest_PIS" name="PIS - sepa - TPP should successfully initiate with mandatory fields only" time="40.285838">* def theBank = bankDetails.UnitedKingdom.Natwest ......................... passed

call read(makePayment) .................................................. passed <testcase classname="io.token.features.adapter.obie.unitedKingdom.Natwest_PIS" name="PIS - sepa - TPP should successfully initiate with mandatory and optional fields" time="38.244523">* def theBank = bankDetails.UnitedKingdom.Natwest ......................... passed

As seen above, there are two different name. What name does Jira Xray consider?

As I mentioned in my old comment, Since the name of the bank in "name="io/token/features/adapter/obie/unitedKingdom/Natwest_PIS.feature"
will be updated in each run, the path will also be changed. Although I run the same test case for each bank, Jira creates a new test case each time because the name has changed.

This is the issue that I need to solve. Thanks

bitcoder commented 2 years ago

Hmm. Well, I think the issue you're facing is in the end because you're calling scenarios from within another scenarios (I'm not sure if that's a recommended practice or not in terms of Karate). In terms of Xray, it will look at the testcase element on the junit report... and uses the classname and name attributes. How Karate generates the junit xml report is beyond of Xray control; I don't know exactly those internals of Karate, so in that respect I'm unable to help, sorry. This doc explains how Xray processes the Junit xml report.

bltemrah commented 2 years ago

@bitcoder thank you. FYI, There are two names in the karate creates XML reports, first is path, second is the scenario name. But, Xray considers the name which includes path, not the scenario name one. This is the problem that I need to solve. I want Xray create test case based on the scenario name, not path name.

<testsuite failures="3" name="io/token/features/adapter/obie/unitedKingdom/Natwest_PIS.feature" skipped="0" tests="9" time="317.715079"><testcase classname="io.token.features.adapter.obie.unitedKingdom.Natwest_PIS" name="PIS - sepa - TPP should successfully initiate with mandatory fields only" time="40.285838">* def theBank = bankDetails.UnitedKingdom.Natwest ......................... passed

bltemrah commented 2 years ago

Hi @bitcoder and @CMCunha Is it possible to send the environment, which is already defined in the Jira-Xray, with/in the below script;

curl -H "Content-Type: text/xml" -X POST -H "Authorization: Bearer $TOKEN" --data @"io.token.features.adapter.nextgenPsd2.germany.Commerz_AIS.xml" "https://xray.cloud.getxray.app/api/v2/import/execution/junit?projectKey=CQA&testPlanKey=CQA-TicketNumber"

curl -H "Content-Type: text/xml" -X POST -H "Authorization: Bearer $TOKEN" --data @"io.token.features.adapter.nextgenPsd2.germany.Commerz_AIS.xml" "https://xray.cloud.getxray.app/api/v2/import/execution/junit?projectKey=CQA&testExecKey=CQA-TicketNumber"

If yes, does the Jira Xray automatically put all test results under the defined environment?

Meantime, is it possible to trigger the test execution by using Jira Xray. As you know, general approach we first run our test cases in our framework, then send their results to the Jira Xray. How about the reverse? I want to first create a test execution ticket in my jira x-ray which trigger my fw to run the test cases.

Thank you.

bitcoder commented 2 years ago

@bltemrah ,

you can pass the test environment using the "testEnvironments" parameter; please check the endpoint documentation here.
The results are not "put" under the defined environment (it doesn't work like that). Test Environments are like an attribute at the Test Execution level, affecting all the results (i.e. the Test Runs) that you report in the context of that Test Execution. More info on Test Environments here. In Xray cloud, you need to have the Test Environments created beforehand btw. Please check documentation.
Concerning triggering, it requires some customization effort depending on the CI/CD tool you use, etc. You can see some examples here.

ptrthomas commented 2 years ago

@CMCunha @bitcoder regarding the proposed change to the JUnit XML output

if you can submit a PR I can include this in the next release

ptrthomas commented 2 years ago

closing this ticket, it can be re-opened with a PR

ptrthomas commented 2 years ago

@bltemrah as mentioned above, I don't see this as a priority as this has nothing to do with karate. many teams are using the junit report in ci/cd with e.g. jenkins. this also hasn't been reported by anyone else. if this has to be changed all this has to be investigated and a PR should address all these concerns

karatelabs / karate

Xray integration #1902