QA Team - Keeping organized reports of relevant Rocket-E2E runs

[MathieuLamiot]

Context

https://wp-media.slack.com/archives/CUKB44GNN/p1726589916414399

We often have failing tests when running Rocket-E2E. While one axis of improvement is to solve those errors, an other one is to keep track of previous Rocket-E2E run reports so that we can check if the errors we face now already occured before (and hence can be discarded). Currently, we have to re-run the tests in old configurations (old plugin versions, old Rocket-E2E versions) to validate or not if an error is accepted, which cost a lot of time. Also, having records will help identify if we make progress or not when it comes to reducing the number of errors we face with Rocket-E2E, and will allow us to identify areas of improvements.

Expected outcome We would need a shared folder or repository to store and access test reports from Rocket-E2E runs. I don't have specific requirements about the setup, but here are some inputs:

• Rocket-E2E is able to output a test report I believe ; we discussed this with @jeawhanlee and @wp-media/qa-team at some point I think.
• We don't need to store all the reports from all the runs we do, but we would need to keep the representative ones (typically when a next version of the plugin and/or the rocket-E2E is released or tagged)?
• We would need a precise naming convention to easily get the version of the plugin, the version of Rocket-E2E (based on Rocket-E2E repo tags ; if they are not used yet, that's a good place to start), and maybe the website tested against?
• Those details should be properly documented for anyone to easily navigate the reports and add theirs if needed.

[MathieuLamiot]

First initiative: https://wp-media.slack.com/archives/CUKB44GNN/p1726638401014909?thread_ts=1726589916.414399&cid=CUKB44GNN

[Khadreal]

I created a draft PR here. What I did is this added a file which can be run after a test and takes an argument which is used to store the test name and eventually push to git To run this command npm run push-report e2ev1.0_wpr_3.17_e2e We can also explore chaining this command to the existing ones we have, something like this "test:smoke": "$npm_package_config_testCommand --tags @smoke && npm run push-report smoke"

We can also explore using a central directory in the remote environment. Would like to get your feedback on this, so I can refine the PR. @jeawhanlee , @MathieuLamiot , @Mai-Saad , @hanna-meda , @Miraeld

[Edit] Pushing to a repo might be a bit more complex since the repo is private, we might have to do with a folder for now

[Khadreal]

As shared here https://wp-media.slack.com/archives/C05NH7JU4S2/p1732266051553669. Wondering if I should move this to block

I've thought of this two options but doesn't look like feasible Option one -- After a test, commit the test report and push to a (private)repo, this is not working because of permission issue(ssh key for the remote environment) Option two -- Move the test results to a location that's available to everyone, currently we can't do that with our account in the remote environment, if you log in as opeyemi you only have access to opeyemi's directory and can't access mike directory. Right now, a bit stuck on this. If the repo is public, option one would work.

[MathieuLamiot]

@Khadreal I can provide a Personal Access Token with write access to the repo if needed. You'd need to check if it can be configured globally for the git commands on the remote environment. We can also go for a shared folder as all users on the remote env. have access to /var. For instance, I created /var/shared/rocket-e2e-reports where everyone can add a file.

Maybe the folder is best to begin with, and then it'd be up to the @wp-media/qa-team to keep the reports they believe are necessary and upload them to git?

Or we could just go with a public repo? But we would still need a login on Github I think 🤔

Maybe the PAT is the easiest. I don't know, up to you! Let me know if you need anything like a PAT.

[MathieuLamiot]

@Khadreal I create a PAT that gives write access to the E2E report repo. You can find it on BitWarden: "GitHub PAT - Remote Desktop E2E Report Upload"

If you are not sure how to set it up globally on the remote desktop env, you might want to ask @Honemo. I think he did something similar with Rocketlabs a few days ago.

[Khadreal]

[update] I went ahead with the shared folder approach, currently it's working, although the code needs some clean up. To run the command you need to provide the directory name as argument. Example is npm run push-report e2eV1.0_WPR3.16.4_smoke This would move the test results to /var/shared/rocket-e2e-reports/e2eV1.0_WPR3.16.4_smoke directory.

Some of the issues with git approach,

• we would have to set up PAT on each environment, git config global ... variable was confined to the username environment.
• We would need to have a submodule in the e2e environment
• Which branch will this be push to ? newly created branch[Which can be an argument] and create a PR after to merge ?

Lastly, the command don't have to be a standalone, we can chain it to some of the existing commands so we don't have to run two commands every now and then.