Data store/cache service improvements needed

[tdmalone]

v0.0.38 added support for AWS being used for the data store/data cache. This means that Slackémon can now finally be used on a ephemeral filesystem service like Heroku.

However, there are issues inherent in this:

• AWS is an eventually consistent filesystem, which theoretically could cause some player and battle data to be lost, overridden, or to temporarily step back in time.
• Slackémon was originally written around using the filesystem as a data store, so it is quite liberal in its opening of and writing to files. While this doesn't seem to cause too much latency when the calls are coming from nearby (eg. Heroku is also on AWS), it can quickly add up AWS costs.
• There is no concept of file locking built in. This is already an issue that exists even when Slackémon is run with its own local data store: a player trying to perform certain actions simultaneously, across different Slack messages or button actions, could potentially find they lose some data or even worse, corrupt their player JSON file if it is read by a local system while it is being written to.

Slackémon currently has separate settings for an image cache and a data cache, they can both be set to either 'local' or 'aws' independently. However, we're utilising the data cache as both a data cache and a data store.

We likely could solve the above issues by:

1. When 'aws' is set as the data cache, also using a secondary local cache to reduce reads from AWS for the life of the current filesystem.
2. Implementing an additional option for the data store, allowing either 'local' or an alternative store such as Redis or Postgres.
   • Slackémon currently uses JSON files to store player and battle data (one per player and one per battle) so it would be ideal, to avoid having to rewrite alot, to stick somewhat to this method if possible. However, perhaps each individual player's Pokémon could be separated into a separate file to prevent single JSON strings from ever getting too big.
   • The method chosen would need to be easy to set up and support. Both Redis and Postgres are simple on Heroku; we would just need to look into Docker options too.
3. Implementing some sort of file lock for the data store, whether 'local' or the above alternative option is in use.

In addition, while in-memory caching is already in use for most data cache reads (eg. Pokémon & species data, move data, item data) and data store reads (player data and battle data), we could also probably reduce writes by ensuring that there's one save to a player file per request (at the moment there are sometimes two especially due to the slackemon_add_xp() function being called while something else requiring a save is also happening, such as catching a Pokémon).

[tdmalone]

I'm currently working on implementing Postgres as an option for the data store, using docker postgres for dev and intending to use the free hobby plan of Postgres on Heroku for prod (allows up to 10k rows).

Redis is another potential option but is only free on Heroku without data persistence, so that's on the backburner for now.

10k row limit for free Heroku prod should be fine providing we can handle large JSON strings - in the current structure, that's 1 per player, and 1 each per spawn instance and battle instance both of which can be pruned.

[Naramsim]

Hey, @tdmalone, I see the project is expanding 😄

Maybe it's time to create a docker-compose.yml. Basically, it is a file processed by Docker Compose (on Linux it is not shipped with Docker) which creates several containers at the same time and orchestrate them. So we could split the project into the following images/containers:

• Apache2 / Nginx (you choose)
• Php7
• PostgreSQL

[tdmalone]

@Naramsim I'm glad you've suggested that! I had a bit of a read into Compose today but it looked too complicated for me to understand at first, so to start development on this I just installed PG and manually linked the containers together. It was a challenge to get the PHP PG extension installed but I got there in the end!

If you could help with the docker-compose.yml that would be fantastic. I would prefer Apache for now but maybe we can move to nginx later - I know it's faster but just trying to manage how much I take on that I haven't used before ;)

[tdmalone]

(We could even look at Alpine Linux maybe to try to reduce the size later, but I'm not sure if that would just increase complexity with getting things working)

[Naramsim]

I would not stick with alpine-linux, and installing php/apache/postgre on it. I would prefer use three containers that are based on Alpine. That are

• php:alpine
• postgres:alpine
• httpd:alpine

So, in the end, we would have a system composed by three small (and secure) containers, which are started automatically with the command docker-compose up -d

What do you think?

[tdmalone]

I think that sounds perfect!

[tdmalone]

Postgres should now be working in https://github.com/tdmalone/slackemon/tree/data-store as an option for SLACKEMON_DATA_STORE_METHOD. I just need to test this on Heroku and then will probably push it to master and bump to v0.0.39.

This partially implements point 2 (Postgres data store ability) mentioned above at https://github.com/tdmalone/slackemon/issues/21#issue-233526916

The rest of point 2 (using Postgres on Docker) will be covered by using Docker Compose.

Point 3 (file locking) is probably the next priority, then point 1 (augment AWS cache with local cache).

Of course, Postgres isn't implemented properly at this stage - we're basically treating it like a filesystem with each row being a 'file' and containing a big JSON string. In future we should completely rewrite this, drop support for a local data store, and use the database properly.

[tdmalone]

• [x] Automatically create database if it doesn't exist yet, for example when run in a new Docker setup
• [x] Data caching appears it might be broken at the moment, may have been a regression caused by the work in this ticket so far, need to look into
• [x] Fix regression with $action not sending through full data anymore
• [x] Verify that slackemon_get_files_by_prefix() is returning in alphanumeric order when the database storage is being used

---

• [x] Docker Compose (see #35)
• [x] Initial file locking work
  • [x] Basic file locking
  • [x] Make locking an optional variable for now, until it is improved
  • [x] User feedback if waiting for a lock to become available
  • [x] Make locking more intelligent to only lock when writing is just about to happen
• [x] Augment AWS cache with local cache

[tdmalone]

Most of the work this ticket asks for is now in master!

A few more things to look into to make file locking more robust:

• [ ] Timeout if lock takes too long to acquire
• [ ] Is it worth implementing a queueing system rather than just the first try after lock is available?
• [ ] Is it worth trying to make file_exists() then file_put_contents() atomic? (Probably possible with DB, but what about local filesystem?) Potentially look into locks with flock() and fopen()'s modes.

[tdmalone]

In addition to the above, there probably needs to be a regular cronned check to expire locks that did not get removed properly - probably after 2 minutes? Or even 1 minute.