mrc-519: simplify queue creation

[richfitz]

This is the next of the set of major changes to the package in order to make it easier to use. Hopefully this is a bit easier to work through than #7

The primary objective of this PR is to replace the use of context for creating of the queue environment with a function that the user provides (currently this is done through the envir method - see test_rrq - but I would be very open to a better name here).

The idea is that we can then write a package that combines rrq + context for use on the cluster, but that using context would not be a prerequisite to using the queue. Note that this does break how we use rrq in hintr, but that should be easy to adapt.

Then there are things that follow as a consequence of doing that change

• The worker config can no longer contain the redis connection information as we no longer have any shared disk where we communicate (note that this essentially fixes mrc-439)
• The worker process logging had to be overhauled for the same reason
• There are no longer two job queues, just the message and the job queue
• The bulk submission features have been temporarily removed (see comment on ab9eb5078b638f39fb61d93543d0bbe2909d35c9 for justification)

Changes that probably could have been split off:

• worker logging that went through context::context_log now goes though the ordinary worker logging
• the key schema no longer starts with rrq: and that is up to the call to rrq_controller (this change because we're no longer hanging the queue id off of a context id)
• There are some whitespace changes, but more to do so I might do a big style PR at some point
• Worker configurations can be listed and read from the controller
• The half-implemented task_count functionality was removed
• docopt is used for the worker script, which is no longer written anywhere by default

Eventually context and queuer will not be dependencies and future work will go through and remove that.

[codecov[bot]]

Codecov Report

Merging #8 into master will not change coverage. The diff coverage is 100%.

@@          Coverage Diff          @@
##           master     #8   +/-   ##
=====================================
  Coverage     100%   100%           
=====================================
  Files          15     14    -1     
  Lines         945    849   -96     
=====================================
- Hits          945    849   -96

Impacted Files	Coverage Δ
R/utils.R	100% <ø> (ø)	:arrow_up:
R/rrq_controller.R	100% <100%> (ø)	:arrow_up:
R/worker.R	100% <100%> (ø)	:arrow_up:
R/keys.R	100% <100%> (ø)	:arrow_up:
R/worker_messages.R	100% <100%> (ø)	:arrow_up:
R/heartbeat.R	100% <100%> (ø)	:arrow_up:
R/worker_config.R	100% <100%> (ø)	:arrow_up:
R/worker_spawn.R	100% <100%> (ø)	:arrow_up:
R/support.R	100% <100%> (ø)	:arrow_up:
R/worker_run.R	100% <100%> (ø)	:arrow_up:
... and 1 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 5808993...bf05ddc. Read the comment docs.

[richfitz]

So thinking now about applying this in hintr. What we need to do is create an rrq_controller set the envir on this to one which sources naomi? And then spawn some workers using worker_spawn?

We will not be spawning workers, but instead handling that with the provisioning. At the moment we have to spawn because of the limitations around the shared filesystem approach