Scalability: move ephemeral state to redis

[knolleary]

Part of #2782

Whilst the work under #2782 has removed most in-memory state handling, there are some pieces that are not suitable for managing via the postgres database.

• API Rate-limiting - state is currently held in memory. The module we use supports connecting to redis out of the box (or a redis-api compatible alternative)
• Instance/Device inflight state - when an instance/device is transitioning between states, we store some state in memory and some in the database. Need to review what, if anything, would be more suited to holding separately.

### Tasks
- [ ] Identify what technology to use (redis, or an alternative given recent licensing changes)
- [ ] Add to our helm charts
- [ ] Setup authentication mechanism
- [ ] Update rate-limiting to use redis if configured
- [ ] Review instance/device lifecycle to see if any other state needs moving in

[hardillb]

Or some suitable fork/alternative (though current new license only impacts SaaS offerings of Redis iirc) e.g.

• dragonflydb
• keydb
• memcache