Love2d Build Automation with itch.io, GitLab, Docker, & love-release #1

Open oniietzschan opened 7 years ago

What The H*ck Are We Going To Do Today?

Hey gamers!

Have you ever had to create distributable copies of your love2d game for various platforms and upload them all to itch.io manually? Isn't this tedious? I've had to do this a ton of times! That's why I figured out how to make computers do this for me. Now whenever I want to push out changes for my game, I can just sit back and drink low-carb tropical punch while The Power Of The Cloud does all of the busy work.

In this article I'm going to figure out how to teach you the techniques that make my system the most effective. When we're finished we'll have a automated system that does the following:

First, you push a new commit to the git repository where your love2d source code resides.
GitLab will fire up a new Pipeline®™ job running a minimal Arch Linux Docker container with love-release and itch.io's butler installed.
The job will execute love-release in order to generate builds of your game for Windows (64-bit and 32-bit) and MacOS, as well as a .love file for Linux users.
The job will use butler to upload all 4 of these builds to your game's itch.io page. They'll be pushed to release channels named after your branch in git.

This sort of automated system is usually called Continuous Integration or Continuous Delivery.

B-but shru-chan... I don't use GitLab, I use BitHub or GitBucket!!

ドンマイ！ Most of what you'll be learning in this article will apply to leading GitLab competitors as well. For GitHub you can use something like TravisCI. BitBucket has it's own Pipelines. You'll need to do a bit of investigation to find out how to wire everything up, but I don't think you'll need to make many modifications to the commands themselves.

Don't ask me why I wrote a tutorial for GitLab on GitHub. I was up super duper late last night and I typed the wrong URL this morning and now I'm trying to make the best of it.

Prerequisite Knowledge

Don't worry if you've never heard of Docker, Pipelines, or any of that other techno dribble-drabble I mentioned earlier. This tutorial isn't going to teach you everything you can do with these tools, but you'll learn just enough to do some cool shit.

This tutorial does assumes that you have the following:

The ability to make a simple love2d game. Understanding the purpose of main.lua and conf.lua is enough.
An itch.io account with a game which you can push uploads to.
A little command line and linux experience.
A basic understanding of git. What are doing on GitHub if you don't have this? Who invited you?

I'm A Fast-Paced Teen From The Born-Mobile Generation And I Don't Have Time To Read This Whole Post

You're in luck! You can definitely use build automation without too much understanding of what's going on. You will mostly just need to tweak some configuration files.

Scroll down to the bottom and read the section with the header "TD;DR".

Let's Do The Damn Thing

Canto I - A Quick Glossary

I'll define a couple tools we'll be scripting today, just to make sure you're able to follow along if you have never heard of them.

love-release: An open source tool which makes it easy to create distributable builds of your game. We specify -W and -M to generate Windows and Mac builds, the .love file comes gratis.
butler: An official tool provided by itch.io which enables you to upload game files to itch.io from the command line. The syntax for uploading our games is butler push directoryOrFile user/game:releaseChannel.

Canto II - Enter GitLab Pipelines

GitLab comes with a built in Continuous Integration tool called Pipelines. Continuous Integration is a software engineering practice which advocates thin vertical slices of functionality which are developed and merged into a mainline branch (ie. master) daily. This kind of practice is enabled by automation.

For this tutorial, we're not so concerned about Continuous Integration as an development ethos. The main thing we'll be focusing is the automation aspect. Continuous Integration automation is generally triggered by pushing commits to the central git repository. (ie. to GitLab.) Tasks which are commonly automated include: unit tests, integration tests, builds, and deployment. Today, we'll be automating the build and deployment of our game using GitLab Pipelines.

Example of what Pipelines looks like

Since not everyone uses GitLab, I'll mention some alternatives for other nifty git webzones. BitBucket has a solution which is also called Pipelines. I don't think GitHub has an integrated 1st party CI tool, but you can use 3rd party solutions like TravisCI or Codeship.

Now, let's take a look and how we can define the behaviour of Pipelines!

The main place where Pipelines configuration resides is in a file called .gitlab-ci.yml. You'll need to create this file yourself at the root level of your git repository. Here's a simple example of a .gitlab-ci.yml file. This file is not representative at all of what we're trying to accomplish, but it will give you an idea of the syntax and structure.

# .gitlab-ci.yml

image: ubuntu:artful

before_script:
  - apt-get install -y -qq lua luarocks
  - luarocks install busted

some-job:
  script:
    - busted --verbose

Let's go through this example section by section.

image: ubuntu:artful

First, we have the image. ubuntu:artful refers to the name of a Docker image here. We'll learn more about Docker shortly, but for now all you need to know is that this defines the starting linux environment which will be used by our Pipeline jobs.

before_script:
  - apt-get install -y -qq lua luarocks
  - luarocks install busted

Next, we have before_script. This is just a list of commands which each job will execute before it starts working through its own commands. Here we're using apt-get to install install lua and luarocks, then we use luarocks to install busted.

some-job:
  script:
    - busted --verbose

Finally, we have a job itself. The name some-job here is totally arbitrary, it could be anything. (Feel free to name this something funny and hilarious like erect-a-dispenser.) As with before_script, scripts just defines a list of commands which will be run. Here we run busted (a lua unit testing tool) in verbose mode.

These aren't the only directives available for use in .gitlab-ci.yml, but they're the only ones we'll be using today. If you're interested in learning more, then you can check out the official documentation, but this is totally optional and not necessary to follow this tutorial.

Even if we only used what we've learned so far, it would be completely possible for us to accomplish our goal of automating our game's build. We could use the script directive to install butler and all of the dependancies needed to run love-release, then script the build generation and upload process of game.

However, we can do better than this. The main problem we'll run into is speed. In particular, installing everything needed for love-release and grabbing butler would generally make up of the majority of the time spent executing our job. Installing the dependencies might take around 2 minutes when creating and uploading our game builds themselves would barely take longer than 1 minute.

The solution here is to start with an environment where both love-release and butler are already available. We can accomplish this using Docker.

Canto III - Create a Docker Image with love-release and butler

Docker is an open source project which provides the ability to create "containers". A Docker container is basically a sandboxed linux environment which runs inside another host OS. A container is very similar to a virtual machine, except it's faster and more lightweight, and is easier to configure, orchestrate, and share. Docker is buzz on the net right now, if you go to your local fairtrade coffee shop I can guarantee that you'll be able to spot at least a couple bearded Node.js hipsters with Docker stickers plastered all over the sleek lids of their MacBook Airs.

Every time Pipelines starts a new job, the first thing that it does is spin up a new Docker container running a linux environment. The specifics of that container depends on what Docker "image" it is based upon. We can specify which image we'd like to use in our jobs with the image directive in our .gitlab-ci.yml file.

We'd like to use an image which has love-release and butler already installed, so that we don't don't need to wait for these to install every time we make a new commit.

Luckily for you, I've already created a Docker image just like this!

Since the central Docker image repository is public, you can actually use it in your own GitLab Pipeline without needing to create a Docker image yourself. You simply need to specify the image exactly like this in your .gitlab-ci.yml:

# .gitlab-ci.yml

image: shru/arch-love-release:latest

It's out of the scope of this article to explain exactly how to build a Docker image from scratch and upload it Docker Hub so that GitLab can access it. However, I will paste the contents of the Dockerfile used to create my image. I've annotated it, so you should be able to follow along.

# Dockerfile

# Base this image off of a minimal Arch Linux image.
# We use Arch because the version of libzip available in the package managers
# of some other distros is often too old for lua-zip.
FROM base/archlinux:latest

# First, update Arch's package manager.
RUN pacman -Syy && \
  # Install love-release.
  yes | pacman -S \
    gcc \
    git \
    libzip \
    luarocks5.1 && \
  luarocks-5.1 install --server=http://luarocks.org/dev lua-zip && \
  luarocks-5.1 install love-release && \
  # Download itch.io butler and save it to ~/bin
  mkdir ~/bin && \
  curl https://dl.itch.ovh/butler/linux-amd64/head/butler --output ~/bin/butler && \
  chmod 755 ~/bin/butler

# Add ~/bin to PATH, so that you can run butler from anywhere.
ENV PATH="$PATH:~/bin"

Again, you don't actually need to do anything with the script above, but it may concern you if you find yourself outgrowing my image. When that happens, I encourage you to check out the official Docker documentation for more guidance.

Canto IV - Scripting love-release to Generate Builds

Our Pipeline jobs will now start with love-release and butler already installed and ready to use. We can now start writing our build script.

The first thing you'll want to do is add a releases table to conf.lua. love-release will read the parameters you specify here when it generates your builds. The purpose of most of these fields should be self-evident, but if you want more information then I'll direct you towards the love-release repository.

-- conf.lua

function love.conf(t)
  -- ...

  t.releases = {
    -- This is the name of the zip archive which contains your game.
    title = 'GameTitle',
    -- This is the name of your game's executable.
    package = 'gametitle',
    loveVersion = '0.10.2',
    version = nil,
    author = 'Yukio Mishima',
    email = 'im.permabanned@poster.net',
    description = nil,
    homepage = 'http://www.quranexplorer.com/',
    -- MacOS needs this.
    identifier = 'gametitle',
    excludeFileList = {
      'README.md',
    },
    compile = false,
    -- I recommend not changing this, as this tutorial assumes it's value will
    -- be "releases".
    releaseDirectory = 'releases',
  }
end

Once you've defined the parameters for love-release, you can go ahead and start generating your builds in your Pipeline like so:

# .gitlab-ci.yml

image: shru/arch-love-release:latest

variables:
  PATH_TO_MAIN_LUA_DIR: src

build:
  script:
    # Create releases
    - cd "$PATH_TO_MAIN_LUA_DIR"
    - love-release -W -M

You'll want to pay attention to the PATH_TO_MAIN_LUA_DIR variable here. love-release wants to be run while your shell is in the same directory as main.lua and conf.lua, so you'll want to ensure that this variable corresponds to the path to these files within your game's git repository. If main.lua and conf.lua live at the root of your repository, then you can commit this variable and the cd command altogether.

Our job doesn't do very much yet though. It'll generate our Windows and Mac builds and our .love file, but they'll just sit there for now. So lame!

Canto V - Obtaining your itch.io API Key

Before we can upload anything to itch.io, we'll need to obtain our API key!

If you've used itch.io's command line tool butler before, you may recall that the first thing you needed to do before you were able to start pushing builds was to authenticate with your itch.io account using butler login.

Our Pipeline job will also need to authenticate with butler in order to push builds. However, we won't be able to use butler login to accomplish this. butler login is an interactive process which requires manual user intervention. Since we're setting up automation to do everything for us, there's no opportunity for us to actually provide the input required to complete login. Even if we could somehow log into our job in order to complete the login process, we would need to repeat this for every commit we make, since every job starts with a brand new linux environment. It can't be helped, login is simply no good here.

So, how will we authenticate, if login doesn't work? Well, it turns out that butler can also by authenticated by setting the environment variable BUTLER_API_KEY.

Each itch.io account has a unique API Key, here's how you can find yours:

Download the latest version of butler for your platform.
Run butler login and authenticate with the itch.io account that you intend to push your game(s) to.
Find your API Key locally. It's location differs by operating system. It will be a 40-character long, case-sensitive, alphanumberic string.

API Key locations:

Windows: %APPDATA%/itch/butler_creds
Mac OS: ~/Library/Application Support/itch/butler_creds
Linux: ~/.config/itch/butler_creds

Once you have your API Key, all that's left is to set it as an environment variable in your job. We could just do this by running this command in your script: export BUTLER_API_KEY=YourKeyHere, but this would be a not-so-awesome idea!

Anyone who has your API key will have the ability to upload games to your itch.io account. So if you set your key in somewhere potentially visible like .gitlab-ci.yml or Dockerfile, then you'd be in serious trouble. You might wake up one morning and find that all of your games have been overwritten with 5-minute-long Twine stories about the challenges of being an asexual, pescetarian, goth teen in a post-structuralist society. I would not wish a fate this cruel upon even my worst enemy, so please be careful with your key!

Anyways, luckily for us, GitLab has a feature called "Secret Variables". If you set your API key as a Secret Variable, nobody will be able to access it from GitLab, not even you!

To setup your Secret Variables, Start at your game's repository page on GitLab, then navigate to Settings, then to Pipelines, then scroll down a little bit until you spot the "Secret variables" header. Now enter your key. It must be named exactly BUTLER_API_KEY, or it will not work.

yoroshiku screenreader-san~

We're now ready to start scripting butler push to upload your game!

Canto VI - Scripting butler to Upload your Video Game to itch.io

So far our script runs these commands:

cd "$PATH_TO_MAIN_LUA_DIR"
love-release -W -M

If we looked at the contents of src/releases afterwards, we would see the following files:

SickTightVideoGame-macosx.zip
SickTightVideoGame-win32.zip
SickTightVideoGame-win64.zip
SickTightVideoGame.love

We could start scripting the upload of each file like so:

# ...
love-release -W -M
# Upload to itch.io
cd releases
butler push YourSickTightVideoGame-win64.zip itchuser/sicktightgame:win64-channel
butler push YourSickTightVideoGame-win32.zip itchuser/sicktightgame:win32-channel
butler push YourSickTightVideoGame-macosx.zip itchuser/sicktightgame:osx-channel
butler push YourSickTightVideoGame.love itchuser/sicktightgame:linux-channel

This will work, but it's a little bit repetitive. If we want to change something like the game's name or the release channel, we'll find ourselves editing the same string in multiple spots. How moderately inconvenient! I can feel the Repetitive Strain Injury already! We're programmers, we can do better!

Here's the complete script which I came up with:

# .gitlab-ci.yml

image: shru/arch-love-release:latest

variables:
  ITCH_USER: dril
  ITCH_GAME: teletubbyshooter2
  PATH_TO_MAIN_LUA_DIR: src
  CHANNEL: $CI_COMMIT_REF_NAME

build:
  script:
    # Create releases
    - cd "$PATH_TO_MAIN_LUA_DIR"
    - love-release -W -M
    # Upload to itch.io
    - cd releases
    - WIN64_FILE=$(ls *-win64.zip)  ; butler push "$WIN64_FILE" "$ITCH_USER/$ITCH_GAME:win64-$CHANNEL"
    - WIN32_FILE=$(ls *-win32.zip)  ; butler push "$WIN32_FILE" "$ITCH_USER/$ITCH_GAME:win32-$CHANNEL"
    - MACOS_FILE=$(ls *-macosx.zip) ; butler push "$MACOS_FILE" "$ITCH_USER/$ITCH_GAME:osx-$CHANNEL"
    # Workaround for issue where butler renames *.love to *.zip.
    - LINUX_FILE=$(ls *.love) ; mkdir love-file-dir ; mv "$LINUX_FILE" love-file-dir
    - butler push love-file-dir "$ITCHIO_USER/$ITCHIO_GAME:linux-$CHANNEL"

You can use this script for your own project, but just remember to change ITCH_USER, ITCH_GAME, PATH_TO_MAIN_LUA_DIR.

In case you're wondering why the Linux upload is split into two commands and looks different from Windows and MacOS: butler has this "feature" where it detects when a file is an archive and diffs the contents in order to speed up the transfer. Unfortunately this feature has the side effect of renaming our .love file to .zip, which is not what we want.

We can circumvent this feature by moving our .love file into it's own directory and uploading that instead. This hack works great and our users won't know the difference!

I'll walk you through one particular part of this script, just so you don't miss one of it's most useful properties.

variables:
  # ...
  CHANNEL: $CI_COMMIT_REF_NAME

build:
  script:
    # ...
    - butler push file "user/game:win64-$CHANNEL"

$CI_COMMIT_REF_NAME is a variable which Pipelines sets for us. Its value will be set to the name of git branch which our commit belongs. This means that if you commit to a new git branch called, for example, beta, then we'll end up pushing to release channels like win64-beta, osx-beta without affecting the files on our *-master release channels. This can be very useful for testing out experimental new functionality — like Zelda's new "Elf" arrows — with a small subset of your players.

Canto VII - はい！できました！

We're done! If you've followed everything up to this point, then you should be able to start pushing commits to your repository and the automation will take care of everything else.

In case you had trouble following or didn't do something quite right, I've summarized everything below.

TL;DR

Add the following love-release configuration to your conf.lua. Feel free to change some of these values if you want.

-- conf.lua

function love.conf(t)
  -- ...

  t.releases = {
    title = 'GameTitle',
    package = 'gametitle',
    loveVersion = '0.10.2',
    version = nil,
    author = 'Oswald Spengler',
    email = 'zuck@facebook.com',
    description = nil,
    homepage = 'http://www.chemtrailcentral.com/',
    identifier = 'gametitle',
    excludeFileList = {
      'README.md',
    },
    compile = false,
    releaseDirectory = 'releases',
  }
end

Create a new file called .gitlab-ci.yml to the root of your git repository with the following contents. Be sure to change ITCHIO_USER, ITCHIO_GAME, and PATH_TO_MAIN_LUA_DIR.

# .gitlab-ci.yml

image: shru/arch-love-release:latest

variables:
  ITCH_USER: dril
  ITCH_GAME: teletubbyshooter2
  PATH_TO_MAIN_LUA_DIR: src
  CHANNEL: $CI_COMMIT_REF_NAME

build:
  script:
    # Create releases
    - cd "$PATH_TO_MAIN_LUA_DIR"
    - love-release -W -M
    # Upload to itch.io
    - cd releases
    - WIN64_FILE=$(ls *-win64.zip)  ; butler push "$WIN64_FILE" "$ITCH_USER/$ITCH_GAME:win64-$CHANNEL"
    - WIN32_FILE=$(ls *-win32.zip)  ; butler push "$WIN32_FILE" "$ITCH_USER/$ITCH_GAME:win32-$CHANNEL"
    - MACOS_FILE=$(ls *-macosx.zip) ; butler push "$MACOS_FILE" "$ITCH_USER/$ITCH_GAME:osx-$CHANNEL"
    # Workaround for issue where butler renames *.love to *.zip.
    - LINUX_FILE=$(ls *.love) ; mkdir love-file-dir ; mv "$LINUX_FILE" love-file-dir
    - butler push love-file-dir "$ITCHIO_USER/$ITCHIO_GAME:linux-$CHANNEL"

Get your itch.io butler API key, then under your repository Pipeline settings in Gitlab, set it as BUTLER_API_KEY under "Secret Variables".
Start pushing some hot new builds, because you're all finished, buddy!

Goodnight, See You Tommorrow!

If you have feedback or can't quite get it working, please leave a comment and I'll see what I can do.

Thumbs up and give it a comment!

Subscribe for more of the hottest gamer content on Internet!

Leave a comment in the comment box if you love too game!

— with such many love, shru-chan

Hey cool post! I love to see more LÖVE users doing these kind of documentations (I'm glad you did it here in Github where I can follow it ❤️ )

I have a few recommendations for your setup which may do some stuff easier! First thing is:

Use Hererocks to install Lua and LuaRocks, this script really makes your life way easier and allows you to install many different versions of Lua simultaneously.

Run Luacheck in your project, there is even a LÖVE standard in there, this will allow you to detect possible errors before they reach production (also there is probably a plugin for your code editor that will lint your code automatically which you can use while you run). Having a linter has benefits like for example if another person makes a PR, the PR will be tested in the CI and the linter will catch small errors before the game actually runs.

I would talk a little about other CIs like TravisCI for Github, CircleCI, AppVeyor for Windows... etc. There are some small differences with them and you could write a few sentences about this (and link to their docs for more info). But that may be out of your scope, I don't know.

Also LÖVE release can distribute to some Linux systems too which would be cool for Butler!

Excellent post, thanks for the info! I didn't know it was this easy to use Butler 😄

@Positive07 Hey, thanks for the feedback.

Regarding Hererocks: I think these kinds of version managers are useful in one's development environment, where one might have to on occasion use different versions of Lua. However, I don't really see what value it would bring to a short-lived Docker container which really only needs to install and run love-release and nothing else. Maybe there's something I'm not considering, but I want to say that it's overkill.

Regarding Luacheck: I'm already aware of luacheck, and I was actually originally planning to adding a quick section about setting up luacheck and busted/luaunit. I ended up dropping that section just to trim down the length of this post, but now I'm reconsidering. I think when I have a moment I'll add some sort of "Extra Credit" section at the bottom of the article, just to tip off beginners on the fact that these things exist.

Regarding other CI Suites: In my defence, I did mention that they exist. I just didn't want to digress too much and bloat the tutorial. I think dropping some links to their documentation pages is a good idea though, so I'll go ahead and add that.

Also, I am aware that love-release can create .deb files. I'm just of the opinion that .love is a more convenient distribution method for Linux. Maybe some people would disagree with this.

Cheers!

Firstly, thank you for this! It's incredible, honestly. But I've got this issue. Is this still supported?

mac

@funkeh You should report the issue on the love-release repo... I took a look and couldn't find the root cause easily, but MisterDA may be able to fix it

@oniietzschan @funkeh Please change the default value for excludeFileList like this:

function love.conf(t)
  -- ...
  t.releases = {
    -- ...
    excludeFileList = {'README.md'},
    -- ...
  }
end

Or, well, anything else besides '' - empty string Otherwise it fails to find any files as it is using substring search (:find) on a file name and an empty string will be found in any string. Instead of erroring on empty file list, it fails on Mac with that cryptic error message. (See MisterDA/love-release#62)

@ASMfreaK Thanks for the catch! I will fix this.

To be perfectly honest, I've never actually gotten excludeFileList to work how I'd like it to. It's supposed to be a pattern search, right? I've tried using it to remove my Asesprite project files, which end in extension .ase like so:

t.releases = {
  -- ...
  excludeFileList = {
    '.+%.ase',
  },
}

However, this has never worked for me, so I just throw in a find -name *.ase -delete.

I'd submit an issue to https://github.com/MisterDA/love-release, but I haven't rebuilt my container image in like 7 months so I'm worried it might already be fixed, possibly... ^^;;

I wanted to exclude all markdown files from the build, so I used something like this at first:

t.releases = {
  -- ...
  excludeFileList = {
    '.md',
  },
}

It's a hack - this will break if there are any files like 'README.md.pdf' which will also be ignored.

Apparently they are using standard Lua string.find function, so patterns work. You can try my ASMfreaK/alpine-love-release docker image. I think rebuilding your image will likely fail, because love-release switched to luarocks v.3, but Arch Linux still uses v.2

This post is awesome! :+1:

I think rebuilding your image will likely fail, because love-release switched to luarocks v.3, but Arch Linux still uses v.2

I have luarocks installed as a rock itself, until the package is updated. https://github.com/luarocks/luarocks/wiki/Installation-instructions-for-Unix#installing-luarocks-as-a-rock

There’s someone looking to maintain the package https://github.com/luarocks/luarocks/issues/946.

I'd submit an issue to https://github.com/MisterDA/love-release, but I haven't rebuilt my container image in like 7 months so I'm worried it might already be fixed, possibly... ^^;;

I guess not…

Very helpful and funny post. Thanks

oniietzschan / blog