search

rust-gamedev / wg

Coordination repository of the Game Development Working Group
514 stars 10 forks source link

[Needed Tutorial] Making CI coverage as intense as possible #30

Closed Lokathor closed 5 years ago

Lokathor commented 5 years ago

Lately I've been having user MaulingMonkey beef up the CI for my crates. To get it right there's a whole suite of scripts to run, config files for each target to use the right linkers, SDKs to download, so many steps. If I didn't have MaulingMonkey doing it for me I'd be in big trouble, totally lost, and I'd probably just give up. And according to them they don't even fully understand what they're doing half the time.

This isn't so good, because you wanna write the game once and be able to get it working on as many platforms as possible. This usually means that you have to be aware of platforms problems as soon as possible so that you don't build up your code base around a design that doesn't work on all systems.

PROPOSAL: We should find people who are experts at various CI services and with various kinds of unusual rust targets (WASM, iOS, Android, etc) and get together a step by step guide of how to get the CI for a project working well.

There isn't a perfect one size fits all solution for every crate. However, if we can gather some solid information on how people can at least build their crates all sorts of places, that alone will be an important step towards games easily targeting as many platforms as possible as quickly as possible.

MaulingMonkey commented 5 years ago

I have wild fantasies of writing yet another cargo subcommand to help automagic all my problems away with this. Might be prone to https://xkcd.com/927/ . Current state of the art as far as I'm concerned is:

Additional issues:

Android

iOS

WASM

Graphics, Consoles

msiglreith commented 5 years ago

For my own projects I setup a few scripts for Azure Pipelines: https://github.com/norse-rs/azure-ci There are more sophisticated examples for example in tower or tokio with multi platform support.

The travis CI setup from Embark looks also quite nice: https://github.com/EmbarkStudios/tame-gcs/blob/master/.travis.yml

erlend-sh commented 5 years ago

@zicklag made this automated build template for Amethyst games: https://github.com/katharostech/amethyst-drone-config

zicklag commented 5 years ago

Here's my experience and thoughts on CI for Rust crates, and games.

Currently I cross-compile Rust crates for Windows, Mac, and Linux from Linux Docker images using Drone ( like the amethyst-drone-config template that @erlend-sh shared above ). I'm partial to Drone and Docker for a few reasons:

The biggest issue with the approach that you can only run tests for Linux target, because everything is cross-compiled from Linux. Drone does support Windows containers, which would allow you to build and test on Windows while keeping the advantage of a reproducible environment that Docker provides. There isn't any such thing as MacOS Docker images, though.

Lots of people use Travis + Appveyor ( like the trust project for building Rust for many platforms ) to cover all platforms because Travis will do MacOS and Linux VMs and Appveyor will do Windows VMs, but I didn't like having to have two different providers with two different kinds of YAMLs to write and understand.

The best option I can find so far that covers all platforms is Cirrus CI which does Linux, Windows, and Mac VMs, Linux and Windows Docker containers, and additionally allows you to bring your own hardware. I haven't tested it yet, but I like what I've seen so far.

I've also tried out the trust template, but for my Arsenal project I required Python to be in the build environment and it didn't look like that was very compatible with the cross project that trust used to cross-compile the crate.

At this point I am happy with how I build the crates, but I still have to figure out how to test them on Windows, Mac, and Linux. I have considered building the crate with Drone, putting the artifacts somewhere that Cirrus CI can get to them, and then testing them on Cirrus CI, but I'm not sure that is the best option. Eventually, I'm going to need to figure out how to run tests, but I haven't gotten there yet.

Also, the process that everybody uses might be slightly or even dramatically different so different things will make sense for different people, but I think that it would be good to get a solid, well-documented, base that people can build on.

MaulingMonkey commented 5 years ago

you can't shell into the Travis servers

Admittedly annoying.

Lots of people use Travis + Appveyor ( like the trust project for building Rust for many platforms ) to cover all platforms because Travis will do MacOS and Linux VMs and Appveyor will do Windows VMs, but I didn't like having to have two different providers with two different kinds of YAMLs to write and understand.

FWIW, travis does support windows images. They're slow to boot (5 minutes) but I have at least one travis-only project because I'm lazy. There's also a "minimal" image (still has gcc and python) you can use: https://docs.travis-ci.com/user/languages/minimal-and-generic/#minimal - which you can use to reduce implicit dependencies, or to add docker support: https://docs.travis-ci.com/user/docker/

Because everybody has access to the Dockerfile and the Drone script that you use to build your crate, the entire build process is documented.

I theoretically like this, but base images seem super opaque to me and often seem to be used to hide the build dependencies. Maybe it's just my docker inexperience showing through here...? E.g. to track down rust:1.35 I need to navigate:

Which is even more annoying than tracking down the travis rust scripts:

zicklag commented 5 years ago

Which is even more annoying than tracking down the travis rust scripts

I can see that Dockerfile being a little annoying to track down, but at least it is fully reproducible and user-controlled as opposed to being built by Travis. If you want to make your own image where all of the dependencies are more in plain sight, you can do that.

I have years of Docker experience, so I'm very comfortable with it. For people who are newer to Docker, it might present more of a barrier than Travis would. I wouldn't know because I'm so used to it.

or to add docker support: https://docs.travis-ci.com/user/docker/

Yes you can use Docker on Travis which would be as good as a solution as Drone if it weren't for the fact that Drone has a UI built for visualizing the running containers, a YAML format that is tailored to using Docker, and it is fully Open Source.

travis does support windows images.

I did see that Travis had a Windows environment. The docs say it is in early stages. That would probably be better than using Travis and Appveyor, but I was liking the look of Cirus's yaml format better and it let you bring your own hardware, which was a plus.

I don't have a lot in its favor yet because I haven't tested it, but that was my initial thought.

Lokathor commented 5 years ago

I have unfortunately never used docker. I don't think it's out of bounds for the type of guidance we want to give, but I think that we should target people who don't know what they're doing, so if Docker is suggested we need to include at least part of a Docker tutorial as well. Perhaps a link to a very good Docker tutorial elsewhere if there's something already written that's appropriate, and then below the link we say "now that you've read that general tutorial, here's a reminder of what specific command you'll use when setting up your rust CI", or something.

zicklag commented 5 years ago

From a getting started perspective, you don't actually need to know anything about Docker other than the fact that it replaces the typical VM image that you would use otherwise. We should just have to point out some images on Docker Hub that would be useful starting points for your builds. So far, for Rust projects, the official Rust and Rust nightly images are perfect.

AlexEne commented 5 years ago

There is some good discussion here, and having some reliable of CI in place, regardless of the purpose of the project is important.

In reality, if you'd do a game, you would want tests to run on actual consoles not just docker images. Providing a guide for that is difficult due to NDAs.

CI is important, but it's not in the scope of this wg to be a central knowledge source on how to set up jenkins/travis/github actions for that purpose and that's why I am going to close this issue.

For other special CI needs there's the other issue with gpu testing on CI #31 that is more gamedev specific.

Lokathor commented 5 years ago

Well, even if you don't ship to console, there's 6 major targets that CI can be setup and used for:

And yes, end users will have to do some testing by hand. However, a large part of the point is to get library developers to have used CI on the major platforms ahead of the time so that when you do go to develop your game there's a minimal amount of fuss instead of hitting an unlimited landmine of problems as soon as someone does testing on Android and it all explodes or something.

But okay

kvark commented 5 years ago

@AlexEne

There is some good discussion here, and having some reliable of CI in place, regardless of the purpose of the project is important.

True. Gamedev is more sensitive to CI issues because it has more interaction with the hardware/platform/drivers than, say, your generic LZ compressor. So CI becomes a pain for our target developers, and thus producing any kind of assisting materials makes sense to have in the scope of this WG.

Perhaps, the closing of the issue was a bit premature?

AlexEne commented 5 years ago

On device ci and the opened issue here are two different things that are separate in my view.

The on device ci is more connected to gpu testing issue already opened.

We can reopen this if you consider we need two CI issues.

In reality CI is a pain regardless of who the target devs are, it's not a rust gamedev only problem.

Rust embedded has to deal with the same issues if not more, rust-wasm wg has to as well, and it's not a rust only thing. Any language the project is in has the same set of issues regarding CI, be it Swift, C++, Java, etc.