openedx / cypress-e2e-tests

Cypress E2E Tests for Open edX applications
GNU Affero General Public License v3.0
15 stars 25 forks source link

use jenkinsfile #33

Closed kashifch closed 5 years ago

kashifch commented 5 years ago

HI @estute

I have created a simple jenkinsfile for my cypress tests, can you take a look and suggest any enhancements/changes.

I tested it by setting up a local jenkins and it seems to be working fine.

There are some areas where I need some guidance

1) I have used a docker image for all pre-requisites of cypress installation, is that a good approach or do I need to figure out all pre-reqs and install these using a script/manually 2) For env vars I have used secret text option, is it a good approach 3) Do I need to perform some actions after the tests are run

Thanks, Kashif

estute commented 5 years ago

Regarding your questions:

  1. I am little confused by the use of Docker in this job. You have a Dockerfile in the repo, but are trying to pull a public image (published by Cypress). If you want to use your Dockerfile for these builds, you should either build it here in the job (which might make your job take longer and is a little redundant) or build it somewhere else and publish it to Dockerhub via the edx-org (we do this for all of the containers we use at edx, like for devstack. I would reach out to DevOps for assistance on this. It will likely mean adding the Dockerfile to edx/configuration so their Jenkins can build and publish it). If you want to use the public image cypress/base:10, I would suggest removing the Dockerfile from this repo, as it could lead to confusion.

Additionally, using Dockerfiles as agents in Jenkins pipelines is currently not great. It requires that the Docker daemon runs on Jenkins master, and all of the containers get downloaded and cached on master. This isn't great for security, performance or disk space issues. It works fine when testing locally, but build-jenkins sometimes manages hundreds of connected ec2 workers and needs to be performant. We eventually want to move towards using Kubernetes to manage Jenkins and workers (this will be great for local development AND live deployment). For now, although it isn't great, I have to suggest the following:

  1. Yes, this is the correct approach. In order to get the credentials into Jenkins, you will have to work with the Developer Experience team (or soon DevOps) to get them loaded into Jenkins

  2. I would recommend archiving/publishing any resulting files from the test run. Does Cypress create a test report? Take screenshots? Generate logs? All of these can be helpful for people who need to debug the test run. You will likely later want to include this job in the release pipeline, which will require you to write Gomatic/Tubular scripts to query the status of this job to see if it is a pass or fail.

kashifch commented 5 years ago

@estute

Thanks a lot for your detailed response, I will make the suggested changes