Qytera-Gmbh / cypress-xray-plugin

A plugin for uploading Cypress test results to Xray.
https://qytera-gmbh.github.io
MIT License
21 stars 7 forks source link

Attach JSON to test execution evidence #314

Closed reify-lucas-mazzini closed 1 month ago

reify-lucas-mazzini commented 2 months ago

Hi! Thanks for the plugin, it solves a lot of issues on our team!

We have an API test execution and the idea is to save the JSON response on the test execution evidence when the test is executed. On the unit test level the framework that the dev team uses has that capability so I'm taking the shot if that's possible to implement here (we're using the screenshot/videos already, works like a charm!)

Thanks!

csvtuda commented 2 months ago

Hi @reify-lucas-mazzini,

glad to hear that the plugin is making your life easier! So just for my understanding, you have something like this:

describe("GET /users", () => {

  it("PRJ-123 returns 200", () => {
    cy.request("GET", "/users").then((response) => {
      expect(response.status).to.eq(200);
    });
  });

  it("PRJ-456 returns all users", () => {
    cy.request("GET", "/users").then((response) => {
      expect(response.body.content).to.deep.eq(["jeff", "bob"]);
    });
  });
});

And now you would like to attach the responses to the test execution as well?


In the current plugin version, this is not possible out of the box (and Cypress does not report the API requests which were dispatched, as far as I know). Do you think something like this would do as well?

import { attachEvidence } from "cypress-xray-plugin";

describe("GET /users", () => {

  it("PRJ-123 returns 200", () => {
    cy.request("GET", "/users").then((response) => {
      attachEvidence("PRJ-123", "body.json", response.body); // maybe there's a way to avoid duplicating the key
      expect(response.status).to.eq(200);
    });
  });

  it("PRJ-456 returns all users", () => {
    cy.request("GET", "/users").then((response) => {
      attachEvidence("PRJ-456", "body.json", response.body); // maybe there's a way to avoid duplicating the key
      expect(response.body.content).to.deep.eq(["jeff", "bob"]);
    });
  });
});

Out of curiosity, which framework does the dev team use for this?

reify-lucas-mazzini commented 2 months ago

Thanks for the quick response @csvtuda!

Yes, the ideal execution would be as you mentioned as an example, making a cy.request() and then attach the evidence inside the same test.

On how to avoid duplicating the key, we implemented an override on the screenshot command that gets the ID from the test and inserts it into the screenshot name, maybe the same logic can be used :) It uses Cypress.currentTest.title to get the test title and then parses the title with the Jira project key and a regex to return the key

Cypress.Commands.overwrite('screenshot', (originalFn, subject, fileName, options) => {
  const jiraKeyPattern = Cypress.env('JIRA_PROJECT_KEY') + '-\\d+';
  const currentTestTitle = Cypress.currentTest.title;
  const isJiraKeyExists = RegExp(jiraKeyPattern).exec(currentTestTitle);
  if (isJiraKeyExists) {
    const jiraKey = isJiraKeyExists ? isJiraKeyExists[0] : currentTestTitle; // Here would be the Jira key
    fileName = `${jiraKey} ${fileName}`;
  }
  return originalFn(subject, fileName, options);
});

The framework that the dev team uses is Kaocha, they have a plugin to upload results to xray but I'm not 100% sure if the plugin is work from the dev team of the framework team 😅

csvtuda commented 2 months ago

On how to avoid duplicating the key, we implemented an override on the screenshot command that gets the ID from the test and inserts it into the screenshot name, maybe the same logic can be used :)

This is an excellent suggestion! I haven't been dabbling with command overwrites a lot, so many thanks for the input 🥇 I will try to implement it using overwrite, too.

If everything works out as intended, the only thing you'll need to do will be an import inside the commands support file:

// support/commands.[js,ts]
import "cypress-xray-plugin/commands";

Alternatively, if you'd like to do other stuff during the overwrite callback and prevent the plugin from overriding existing custom commands, I will also make sure to expose the handler:

// support/commands.[js,ts]
import on from "cypress-xray-plugin/commands/handler";

Cypress.Commands.overwrite("request", (fn, options) => {
    // Existing overwrite logic...
    // ...
    return on("cy.request", fn, options);
})

I will notify you when the new version is released.

reify-lucas-mazzini commented 2 months ago

Thanks for all the work and the quick responses! ❤️

csvtuda commented 2 months ago

Hey @reify-lucas-mazzini,

sorry for the delay, writing tests for this issue turned out to be a lot more difficult than I had expected.

I just released version 7.0.0, which includes the new command override. For a list of breaking changes, please see here.


As for your issue, you should now be able to add cy.request JSON upload by performing the following steps:

  1. Import the overwritten command to your support file:

    import "cypress-xray-plugin/commands";

    If you want to keep existing custom cy.request, please check the documentation. It includes two examples for this very use case.

  2. Enable the requests upload in the configuration:

    // ...
    async setupNodeEvents(on, config) {
      await configureXrayPlugin(
        on,
        config,
        {
          // ...
          xray: {
            uploadRequests: true
          }
        }
      );
    }

Afterwards, you should be able to see request evidence for tests you've linked to existing Jira issues (issue key in test title):

grafik

The documentation examples also show how you can customise evidence filenames.

Let me know if the new version works as expected and/or fixes your issue.

reify-lucas-mazzini commented 1 month ago

Hey @csvtuda! Thanks for all the hard work, and sorry for the delay on the response, just got some capacity to tackle this!

This are the findings on the update so far:

Updated the plugin, configured everything and got the following error: The 'task' event has not been registered in the setupNodeEvents method. You must register it before using cy.task() "Fixed" it the following way (not sure if the correct way to solve it):

// cypress.config.js

async setupNodeEvents(on, config) {
  on('task', {
          'cypress-xray-plugin:task:request'() {
              return null;
          },
          'cypress-xray-plugin:task:response'(){
              return null;
          }
  });
  await configureXrayPlugin(
    on,
    config,
    {
      // ...
      xray: {
        uploadRequests: true
      }
    }
  );
}

After solving those issues, the cy.request() function it seems to be not working at it should.

// cypress.test.js
      cy.request('https://jsonplaceholder.typicode.com/comments').then((response) => {
          expect(response.status).to.eq(200)
      });

Using the command as normal gives the following error:

cy.request() requires a url. You did not provide a url.

image
csvtuda commented 1 month ago

Hi @reify-lucas-mazzini,

Did you import the plugin's commands in your support file? E.g.:

// ...
// Import commands.js using ES2015 syntax:
import "./commands";

import "cypress-xray-plugin/commands"; // <<<<<

The 'task' event has not been registered in the setupNodeEvents method. You must register it before using cy.task()

In version 7.0.0, the plugin does so when calling configureXrayPlugin: https://github.com/Qytera-Gmbh/cypress-xray-plugin/blob/a106ecbe383535f28dc9fed05d50a56ef9d218a5/src/plugin.ts#L84

I don't understand why Cypress is complaining there.

cy.request() requires a url. You did not provide a url.

This is because you're returning null inside your "custom" tasks. The minimal configuration I've used during development consisted of the support file above as well as the following configuration:

// cypress.config.js

async setupNodeEvents(on, config) {
  await configureXrayPlugin(
    on,
    config,
    {
      // ...
      xray: {
        uploadRequests: true
      }
    }
  );
}

If you want to, I can set up a public demo project with a working configuration.

Edit: Can you check if everything works fine when using npx cypress run instead of npx cypress open? It might be related to that.

reify-lucas-mazzini commented 1 month ago

Hey @csvtuda!

1- I'm using "cypress-xray-plugin": "^7.0.0" for the package version. 2- Yes!

Trying to run both scenarios (with task "fix" setup on the config file and the url) with npx cypress run and got the same issues:

image image

The issue here is that without the task configuration, the first error is thrown. Even if I set the return on the task to true it returns a different error.

image

I already try to uninstall the package and installing it clean, any recommendations on how to proceed? 👀

csvtuda commented 1 month ago

Hey @reify-lucas-mazzini,

I created a sample project for your use case here: https://github.com/Qytera-Gmbh/cypress-xray-plugin-examples/tree/main/request-upload. As stated in the README, everything works fine in that project.

Can you compare that project with your current one and try to see the differences? Feel free to clone it, the README should contain all the steps necessary to run it.

reify-lucas-mazzini commented 1 month ago

@csvtuda sorry for the late reply! Got stuck with other tasks and another person from the team picked this up.

We got it working! It was something with my local (we still don't know what was it :/)

Thanks for all the support on this, very appreciated all the work on the package ❤️

image