search

cue-unity / unity

unity - run experiments/regression tests on CUE modules
Apache License 2.0
41 stars 5 forks source link

consider running project tests in operating systems other than Linux #109

Open mvdan opened 2 years ago

mvdan commented 2 years ago

Right now, in the default "safe" mode, we run tests inside a Docker container, where the image is currently https://github.com/cue-unity/docker. This means that, for now, we only run downstream tests on Linux. If a developer runs Unity on Mac, for example, Docker for Mac will use a Linux VM to run the containers.

This is probably fine for the foreseeable future, as the majority of CUE configs and workflows should be OS-agnostic. And Linux is probably a good default to use for now, as it's often the most common OS to use for CI/CD and cloud/server deployments.

However, some downstreams might want to instead (or also?) test on other operating systems. For example, I could imagine a downstream project where CUE is used in a company that mainly develops on and for Windows. Their configs would likely reflect that, and they might not care about supporting Linux at all. In terms of Unity's regression testing, we should be running their tests on Windows.

This issue is a placeholder and reminder about supporting other OSes. Windows and Mac aren't free to use, so we would likely need a different setup to run their tests other than local Docker containers, such as using the free Mac/Windows runners on GitHub Actions.

cc @myitcv

mvdan commented 2 years ago

I should also note that Linux and Mac are often similar enough (e.g. path separators) that it's likely that we can get away with not supporting Mac for some time. Windows will likely be different, as it is very different from Linux in many ways.

4ad commented 2 years ago

There are two orthogonal aspects to this, I think.

  1. The ability to choose your own runner.

Unity now is hardcoded to use docker, but, for example, maybe I don't want to run docker on macOS, or on Windows there is no docker at all. Or maybe on Linux I want to run podman instead of docker, etc.

  1. In the actual CI system use the OS/runner combination that the users expect.

This issue is about the latter, but the issues are orthogonal, There's no reason one should not be able to run the "Linux tests" on Windows without docker.

mvdan commented 2 years ago

Unity now is hardcoded to use docker, but, for example, maybe I don't want to run docker on macOS, or on Windows there is no docker at all. Or maybe on Linux I want to run podman instead of docker, etc.

One clear option is "run directly", which is what the current "unsafe" mode does. This is likely what we will want to be able to run downstream tests on Mac, as I don't think it has anything akin to containers.

Aside from that, or alternative docker implementations like podman, is there any existing standard which is as widespread as Docker? I can't think of anything like Docker container images other than VM images, and we likely don't want to use full VMs. What we need is closer to a chroot.

There's no reason one should not be able to run the "Linux tests" on Windows without docker.

Sure, though they might fail depending on how much they depend on a Linux-like host system.

4ad commented 2 years ago

Aside from that, or alternative docker implementations like podman, is there any existing standard which is as widespread as Docker?

Well podman should be an (almost) drop-in replacement for docker[1] (RedHat distros ship with docker=podman alias, IIUC), but of course, unity needs to know to run a different binary.

[1] I believe there might be some minor differences regarding filesystem sharing.

myitcv commented 2 years ago

Drive-by comment:

  1. The ability to choose your own runner.

Not only that, but the equivalent of what Docker image to use with that runner. I think our schema for cue.mod/tests probably needs to be aware of such things.