Optimization: Invariant payloads

[atruskie]

As detailed in: https://github.com/QutBioacoustics/baw-private/issues/110#issuecomment-152883850

Enqueuing a large batch of jobs (330K) creates as many entries in Redis. Currently, the entire job payload is duplicated even though 90% of it does not change. In particular, verbose configuration files (like in the Towsey.Acoustic.yml case) take up 3715/4239 chars, 88% of the payload. For 330K - 1 jobs, 3715 bytes are redundant; that is 1.23GB of unnecessary RAM use on a redis instance.

Investigate methods of storing the bulk of the payload (especially the config strings) only once. I'm thinking a redis hash string value where the key is the SHA of the config, and the value is the config {{namespace}}:{{object-type}}:{{id}} and the value is the JSON encoded rest of the payload. Jobs then do two redis operations, one for the payload (done by resque) and one for the config rest of they payload (the invariant payload). An example key may look like: baw-workers:analysis_job:123 or baw-workers:analysis_job:system:index_genration.

Important requirements:

• must be able to work with a full payload or a partial payload
• must be able to work for system jobs or analysis jobs
• invariant payload creation can be invoked elsewhere
  • Analysis jobs can delete their payloads when created and finished.
  • System jobs should periodically update their invariant payload
• the invariant payload should have a reasonable TTL - something like 3 months.
• a normal payload will have a special key in it's JSON string. The key, named payload-base, will contain the absolute redis key to the rest of the payload. E.g.

{
  "uuid": "7d78905e-5ea9-4a95-b568-0f14c2f37e4c",
  "id": 1234,
  "payload-base": "baw-workers:analysis_job:123"
}

[atruskie]

Okay further investigation into this issue is revealing some super interesting information.

Redis is currently using 4GB of RAM when idle.

Using https://github.com/gamenet/redis-memory-analyzer and https://redisdesktop.com/ I've worked out the following:

• 99% of they keys stored are from the resque:status gem. That's 545073 keys; one per resque job since Feburary this year.
• all of those stored are from completed (or failed) jobs!
• the string values for the resque:status gem's keys are using 3.4GB or RAM, and 4.5GB in reality (1.33 ratio overhead)
• The TTL for those keys is set to -1 which is why they do not get deleted
• The resque:status gem stores the options (the payload for the job) as well as serializing the entire payload again for use as a 'friendly' name field (2x memory usage!)
• For failed jobs:
  • The resque:status gem stores the payload the same as every other status
  • The resque:status gem does not store exception data from the failed queue. It appears the resque failed queue is completely isolated and only it stores exception data.
  • For resque failed queue (it is a LIST) for 7000 failed jobs uses 111MB of RAM (stores full payload, and excpetion stack trace)
  • conclusion: resque failed queue itself is not a problem
• see https://gyazo.com/335a2981636d995da4cb17e611e0be3a

New issues created:

• 41

• 40

All of these effects are issues with storage after jobs have run, the partial payload idea may be still useful because enqueuing a massive amount of jobs will still chew up memory. I plan to fix those bugs first, profile performance, and then complete partial payloads if necessary.