medic / cht-core

The CHT Core Framework makes it faster to build responsive, offline-first digital health apps that equip health workers to provide better care in their communities. It is a central resource of the Community Health Toolkit.
https://communityhealthtoolkit.org
GNU Affero General Public License v3.0
467 stars 217 forks source link

Create new commands that would build the docker images and run the e2e tests separately #9630

Open tatilepizs opened 5 days ago

tatilepizs commented 5 days ago

Describe the issue When we are creating new e2e tests or debugging the existing ones (and not modifying the cht-core code), it is really frustrating that every time we run the e2e tests we need to download and build completely the docker images. It takes around 2 minutes for a test to start running, which is a lot of time when we are debugging, for example, and just need to try small things in each iteration.

Describe the improvement you'd like We would like to create two new commands, one that builds the docker images and one that only executes the wdio e2e tests.

The first command, the one that build the images, would be in charge of deleting always the images associated at the branch (if they exist) and then creating them. if we remove the ${buildTime} from this line we can use the same images every time that we run the tests using the same branch, and by deleting the images at the beginning we would make sure that we are using the correct images, in case something changes in the cht-core code. The second command would be in charge of only executing the e2e tests, and since everything will be already built, it should just take seconds to start seeing the execution of the test.

We will also leave the command wdio-local that would build and execute, just as it is right now.

Describe alternatives you've considered

tatilepizs commented 5 days ago

@dianabarsan, I would love to hear your thoughts about this idea. Do you think that the approach that I am describing is feasible? Is there something else that I should consider?

cc: @jkuester

dianabarsan commented 5 days ago

I think I already had a branch that did this. I'll share.

dianabarsan commented 1 day ago

Ok, it's not a branch, because you don't need new commands.

All you need to do is:

jkuester commented 14 hours ago

@dianabarsan is there a compelling reason that we need to timestamp the image version? The main benefit I can think of is that it helps ensure we don't locally run the e2e tests against a stale version of the docker image. Is there something else I am missing?

The downsides of timestamping the image version include:

IMHO, re-running the e2e tests without rebuilding the images should be a workflow with top-level support. As a developer I am frequently re-running tests while writing them. It would be easier to script this if the image VERSION was deterministic (and not time-stamped). However, even if we keep the time-stamping, I still think it would be valuable to have an npm script that would programatically set the VERSION when necessary.

dianabarsan commented 3 hours ago

re-running the e2e tests without rebuilding the images should be a workflow with top-level support

Sure, I agree.

If we neither timestamp the images NOR rebuild the images every time we run tests, then the risk of running tests over unknown images is really high. How do you even know what's in the image? Do you start the container and inspect it to see what changes are in it? Especially if you run multiple git worktrees and run tests in parallel.

This is one argument for timestamping, that if one uses different worktrees (like me), then you will have different images for them - which you can reuse with the VERSION env. Now, the worktrees don't necessarily need a timestamp (though this is most correct), we could use a hash of the local folder path instead to uniquely identify the reusable worktree images.

It's a shame there is no "only build if changed" docker command :(

I still think it would be valuable to have an npm script that would programatically set the VERSION when necessary.

What can be done is:

export VERSION=mytest
npm run integration-all-local
npm run ci-integration-all
npm run ci-integration-all
npm run ci-integration-all
npm run ci-integration-all
npm run ci-integration-all

This way you don't need to fish around get timestamps from generated docker compose files.

dianabarsan commented 3 hours ago

What if .... instead of timestamps we actually used the local checked out branch name? That way the version is deterministic, we can include it in npm scripts and it will also work with different worktrees or when working on multiple branches in parallel?

VERSION=$(git branch --show-current) npm run integration-all-local
VERSION=$(git branch --show-current) npm run ci-integration-all

And these can be our npm commands for local runs.