Continuous integration and automatic releases with Travis CI

[rgov]

I noticed that you were unable to release a macOS version of 0.19.1 due to hardware issues, so I tried to see if I could use Travis CI to automatically build it for you. It turned out to be pretty difficult!

This pull request adds a .travis.yml file that teaches Travis to install Unity 2018.2.0f2 and subsequently build Armok Vision for macOS, Windows, and Linux. Finally, it can also automatically upload the build artifacts to GitHub.

Note that you will need to make some changes after this has been merged in.

• [ ] You will need to replace the encrypted configuration fields that are specific to my fork. You will need the travis command line tool.

  • [ ] Encrypt your GitHub access token. I would suggest running travis setup releases --force; the answers don't matter except for Encrypt API key? yes. Then grab the api_key from .travis.yml, discard all of the changes to that file, then paste it in yourself.
  • [ ] Encrypt your Unity account e-mail with travis encrypt UNITY_USERNAME="foobar@example.com" and paste the result into .travis.yml where noted.
  • [ ] Likewise, encrypt your Unity account password with travis encrypt UNITY_PASSWORD="xyzzy".

Note that Travis CI attempts to sanitize these sensitive values from build logs, but they could be accessed if Travis CI is hacked. If you're paranoid, you could set up a separate account for Unity, but make sure you go through the activation process at least once on your own machine.

• [ ] Once that's ready, you need to activate this repo on Travis CI.

Automatic releases work like this: If you push a new tag, Travis CI will build that tagged version and then create a new release with the same name as the tag, with the tagged commit message as the description, and attach the built binaries.

[rgov]

Please review and prepare to merge (i.e., go ahead and generate encrypted secrets) but do not merge until I've had a successful end-to-end build on my forked repo. So far I've tested:

• [x] Installation of Unity works
• [x] Activation of Unity works
• [ ] Build completes successfully → last build timed out, increased timeout to 90 minutes
• [x] Deployment as a GitHub release works

[RosaryMala]

Nice. I'll give this a go once a successful build is done.

[rgov]

One issue is that Travis limits each job to 50 minutes. I split the build into 3 parallel jobs, and the Windows build completed in 43 minutes 26 seconds, but the other two jobs timed out.

Is there anything that can be done to optimize the build time? Even a 10% speed up might make this possible.

There is another potential issue. I'm not sure that Travis can deploy artifacts from multiple build jobs into a single GitHub release. If this is the case, there are a few solutions:

• Only build and upload the Mac version on Travis, and then rely on you to manually build and attach the Windows and Linux binaries to the release that Travis creates.

• Temporarily upload each build product to Amazon S3, and then upload them all to GitHub after the last job finishes. This would have some negligible associated cost.

[RosaryMala]

Can a job from travis be triggered manually?

[rgov]

Yes, see here, but I don't think it would address any of the problems. It's best to automate it, anyhow.

[RosaryMala]

Probably best would be to have one travis job for each OS on release.

[rgov]

Yes, that's what bac3724 did, but the Mac and Linux builds take too long. See https://travis-ci.org/rgov/armok-vision/builds/409278818 (if this is a public link, not sure).

[rgov]

Have you tried Unity Cloud Build as an alternative? Maybe we could figure out how to get it to upload to GitHub instead of using Travis CI.

[RosaryMala]

I used to use that, but it's not free anymore, I believe. Also it can't upload builds to GitHub automatically, as far as I know.

[lethosor]

The GitHub API supports uploading assets individually. What's the issue with doing that in Travis?

[RosaryMala]

It's not Travis that's the issue, it's Unity cloud build

[lethosor]

I was referring to https://github.com/JapaMala/armok-vision/pull/71#issuecomment-408610481, which I'm pretty sure is about Travis.

Also, it's worth noting that Travis build times can be very inconsistent. For instance, times on https://travis-ci.org/DFHack/dfhack/builds range from 8:41 to 13:41, and didn't contain any changes that should affect build time much. I don't think it's safe to assume that Windows builds will always be 43 minutes and not under 50.

[RosaryMala]

I'm not really sure what can be done to decrease the built times, to be honest, unless Travis can cache things between builds.

[rgov]

It can cache between builds. What would you propose caching? And what % of build time does it represent?

[RosaryMala]

There's a lot of stuff, particularly shaders, that doesn't need to be re-compiled often. If it can be cached, that cuts down a lot of the built time.

[rgov]

Here's the Travis caching documentation: https://docs.travis-ci.com/user/caching/

Before the build, we check if a cached archive exists. If it does, we download it and unpack it to the specified locations.

After the build we check for changes in the directory, create a new archive with those changes, and upload it to the remote storage.

Do shaders need to be recompiled once per target platform (macOS/Win/Linux) or just one time? How can we detect whether the shader cache is out of date, and needs to be rebuilt?

My thought here is that we could add a "Rebuild Shaders" stage that happens first. Ideally, there would be a short-circuit for skipping rebuild, for instance if we can tell that none of the shader source files have been modified. If rebuild is necessary, then we install Unity and invoke some method like RebuildShaders(). Finally, Travis uploads the new built shader cache.

Then the second stage of "Build Armok Vision" fires up with a parallel job for each platform. Each job downloads the shader cache. It installs Unity, then invokes Build<Platform>(), which must be modified so that it does not rebuild anything in the cache.

If you can make the necessary build process changes, I can wire them up to Travis.

[lethosor]

One problem with that is if the build times out, the cache won't be updated. But if you want to go with that strategy, you could probably just store hashes of the relevant files in the cache too.

Is it possible to do partial and incremental builds here? If so, you could simply split the build process into multiple commands.

[RosaryMala]

https://docs.unity3d.com/Manual/CacheServer.html

This would be something to look at.

[RosaryMala]

I've activated a pro unity services account, with space for 3 team members. We can use that for cloud builds.

[rgov]

One problem with that is if the build times out, the cache won't be updated.

The cache rebuild would be a separate job that would have a full 50 minutes, and I don't think it would take that long.

Is it possible to do partial and incremental builds here? If so, you could simply split the build process into multiple commands.

Not really, you'd have to provide your own storage for passing artifacts between phases. Travis docs

I've activated a pro unity services account, with space for 3 team members. We can use that for cloud builds.

Well, I guess we don't need this pull request anymore. Shall we close it?

[rgov]

If you close it, please still update the build instructions in the README for the correct Unity version to use.

[RosaryMala]

Since you probably know more about the GitHub API than me, would you like to be added to the Unity cloud build permissions, so you can help see if automatic uploads can be done?

[RosaryMala]

So far as i can see from some initial browsing, it would require a server to do the middle work between both Unity and GitHub

[rgov]

What is your goal here? It was basically a freebie in Travis CI to get it to upload binaries for you, but if you're not planning on making releases often, I wouldn't worry about automating it, and just upload stuff to GitHub Releases manually.

You can still use Unity Cloud Build to build every commit (docs) but it doesn't necessarily integrate with GitHub as nicely as Travis, showing you the status of pull requests and so on.

[RosaryMala]

Well i pretty much never get build requests anyway, so....

[lethosor]

The cache rebuild would be a separate job that would have a full 50 minutes, and I don't think it would take that long.

No, if a single command times out, the build is terminated with no chance to update the cache. I've seen this happen in DFHack builds before.

Not really, you'd have to provide your own storage for passing artifacts between phases. Travis docs

No, I'm not talking about separating it across jobs. I'm talking about splitting the single build command into multiple commands. My understanding from reading the Travis docs yesterday is that individual commands time out, but two commands that run for e.g. 30 minutes each are fine.

[rgov]

No, Travis docs say that a job times out in 50 minutes. A job is a collection of commands run within a single virtual machine. You can split the build into multiple jobs, all run in separate virtual machines, as long as every job takes under 50 minutes.

It sounds like the cache is updated even if the build fails so that's not a problem.

[lethosor]

Ok, I was misremembering what I read yesterday. I'd seen an individual command time out before, but that was probably because it timed out due to not producing output for 10 minutes.

https://travis-ci.org/lethosor/dfhack/builds/405376907 is the build I was referring to, and I'm not seeing any indication that it wrote to the cache. Allowing terminated builds to write to the cache is probably a bad idea anyway, since they could write incomplete things to the cache that would cause issues with later builds.

I'm pretty sure the link you gave means the "failed" state, when a build finishes but does so unsuccessfully. When a build times out, it goes into the "errored" state, which is different and doesn't upload the cache, at least as far as I can tell from the build I linked.

[rgov]

If the Unity Cloud Build is working, can you attach a built copy of 0.91.1 for macOS now?

[RosaryMala]

Done. I'm closing this, because it isn't working.