Containerize!

[Hoikas]

In the interest of making it easier to fire up MOUL servers for development and testing of both code and ages, this adds support for firing up DirtSand using docker-compose. The end goal is for fan-age creators to actively use the superior MOUL client with its debug tools locally instead of the legacy PotS engine.

The general procedure is to copy dirtsand.sample.env to .env and edit the path variables contained therein. These paths will be mounted as bindpoints on the host OS. The dirtsand container will generate the server.ini for the client at startup and use the age/sdl files provided in the respective mount points. Currently, if no age or sdl files are present, they are copied from the H-uru/Plasma master files fetched at image build time.

---

While this works "as-is" currently, it's not quite as easy to use as I would like, and I have some questions about how this should be approached:

Should this even be in the DS repository?

Firing up a fast container is nice and, I think, a needed feature for modern servers. However, there is still some grunt work that has to be done as far as: manually downloading the moul-assets, compiling a client, grabbing the python files. copying the server.ini, etc. Would it make more sense for this to be in a separate repository with perhaps powershell script(s) to automate these tasks? I'm not sure PS scripts belong in this repository.

Age/SDL file handling

Presently at image build time, we grab a copy of the latest H-uru/Plasma scripts and use them "just in case" they are not provided at startup. This is a nice ease-of use touch, but there are still other sticking points as mentioned above. Would it make more sense to just error out at startup if those files are missing?

I'm sure there are other concerns that I can't quite put words to yet. Feedback is appreciated!

[dpogue]

All the encryption keys are blank in both dirtsand.ini and server.ini when I run this

[dpogue]

Could we take in a default account email and password from the env file and automatically run ds acctadd?

[Hoikas]

The only issue I have with that is the need to continually check the env contents and update that account's name/password. It could be done though.

[Hoikas]

I've updated the dirtsand Dockerfile to use alpine as the base image. I'm not sure how to compare the resulting image sizes, but the alpine base download and apk downloads seem faster than what was being done with the gcc image.

[Hoikas]

Updated to track postgresql 12 due to from feedback by @dpogue that postgresql 13 breaks something or another.

[Hoikas]

So, I apparently never marked this thing as "ready for review." I have no idea why not, so I am doing so now. :trollface:

[zrax]

I'm curious why you put the db in a separate container... Alpine also has postgres server packages; wouldn't it be sufficient to put everything in a single container?

[Hoikas]

My understanding is that, in general, containerized applications are supposed to follow the general rule of "one container per service."

[Hoikas]

OK, I have addressed a few different aspects of the feedback, with some caveats.

I still do not think that we should run postgresql in the base dirtsand container. The two containers perform two inherently different jobs and ought to remain separated, IMO. Besides, someone may want to run the DB on bare metal and just dirtsand in a container - most Linuces have postgresql packages, after all. We should not force anyone into a particular stance on this issue by limiting the dirtsand image. That being said, I have removed the dedicated db container and converted it to a postgresql:13-alpine image in the docker-compose file. This image is ~60mb compressed and 160mb on disk. To facilitate this, I ensured that dbinit.sql is idempotent. Further, I have corrected the issue by which the dirtsand image was excessively bloated. Previously, it was ~220mb on disk. Now, it is 19mb.

I have also made the whole system easier to use. Previously, there was a strange .env file with random settings that needed to be tweaked. This file no longer exists. For the purpose of starting up a singleton dirtsand instance for development purposes, I have added the scripts dockersand.sh (for Linux and WSL) and dockersand.ps1 (for Windows). To set up a local shard, it is now sufficient to:

git clone https://github.com/H-uru/dirtsand.git
dirtsand/dockersand start
cp dirtsand/build/etc/server.ini path/to/plClient/dir
dirtsand/dockersand attach
ds-918> addacct <username> <password>

then run plClient.exe with /LocalData

I have converted this to a draft because, currently, this is only suitable for hosting local development shards. All the database credentials are hardwired in docker-compose.local.yml. This is OK because only the dirtsand container should be able to talk to the database container. In a real shard, however, we'll need machinery to expose the db container to e.g. an nginx/php container or to the host machine. Meaning the credentials need to be configurable and secured. This isn't hard, it just requires upfront work by the person deploying the server.

Even though this is a draft, please do review the changes! I think this is a huge improvement overall and will be the start of a system Age creators can use to test their content online.

[zrax]

I still do not think that we should run postgresql in the base dirtsand container. The two containers perform two inherently different jobs and ought to remain separated, IMO. Besides, someone may want to run the DB on bare metal and just dirtsand in a container - most Linuces have postgresql packages, after all. We should not force anyone into a particular stance on this issue by limiting the dirtsand image. That being said, I have removed the dedicated db container and converted it to a postgresql:13-alpine image in the docker-compose file. [...]

It makes sense; I'm still learning Docker as well... Did you say there were issues with postgres 13 though?

[Hoikas]

I got rid of the duplicated compose files; they were dumb. Anyone who wants to deploy a shard should write their own compose file and tweak dockersand.sh.

Did you say there were issues with postgres 13 though?

@dpogue reported a problem with it a few months ago. I think the issue was that his db container updated itself from PostgreSQL 12 to 13 because the version wasn't explicitly tagged in the Dockerfile. When it loaded the old database volume, Postgres panicked on seeing old database files. As long as we keep the db container tagged at a specific PostgreSQL version, we should be good.

[Hoikas]

Ok, I am stumped. When using this on real Linux(TM), the moul service fails with a permission denied error on the etc bind mount.

[cwalther]

Does this help? https://www.eclipse.org/lists/cdt-dev/msg34852.html

[dpogue]

Firstly, dockersand.sh needs to be marked +x to be executable.

It looks like it's trying to attach /home/dpogue/Coding/dirtsand/build/SDL to the container, but my dirtsand folder has no build directory.

The complete set of steps I ran:

git checkout docker        # From master
chmod +x dockersand.sh
./dockersand.sh status     # "Dockersand is DOWN"
./dockersand.sh build
./dockersand.sh status     # "Dockersand is DOWN"
./dockersand.sh start      # bind error

[dpogue]

So it seems to work fine if you create the build folder structure before running any of the dockersand commands:

mkdir -p build/{SDL,dat,etc,fileserv,authserv}

[Hoikas]

Thanks for the suggestions, everyone. I've moved dockersand to a multi-stage build as suggested. As a bonus, I used this as an opportunity to make a configuration where it's easy to debug the server. Just change the BUILD_TYPE to debug in docker-compose.yml and rebuild the container - then you can docker exec -it dirtsand_moul_1 gdb at any time to debug server crashes. There should also no longer be any need to manually create any directories or chmod any files.

[Hoikas]

🎉 I will have to add some documentation now, I think,

[ZarothYe]

With the addition of setup documentation and a couple tweaks to smooth out some issues and confusion I ran into, this feels ready to me now. I was able to set it up and get going pretty easily, and Briggs has reported so as well. Would it be possible for @Deledrius @dpogue @zrax to give this a once over and see if we can approve this?