Multi-chalk awareness for single point of communication

[drraid]

Description

Multiple instances of chalk could be running on a host (or in the container case, in the same namespace(s)), and support for awareness and communication between them is necessary for:

• synchronization of heartbeats, combination of output, such as if three containers run in the same pod and all have the same configuration for heartbeat, they should report as one to both give the user better context and avoid resource exhaustion from thundering herd type scenarios

• multi-process monitoring and reporting, which for now we could contrive as necessary from concurrently running docker builds (again with the same chalk profiles), but there's a bigger picture vision of using Curiosity to instrument builds which will then involve subprocess monitoring and communication between concurrently executing programs

Thus we need chalk to have:

• Approach to establish a leader process per-chalk-config to own off-host communications, open a channel for local communication with other subordinate chalk processes (author suggestion is flock() on unique config/profile-derived unix socket path)

• Ability to discover that a leader process exists for a given chalk config, send it

• Protocol for what chalk needs to communicate to leader process, how leader process consumes and appends it, and how it ships the data it consumes (in addition to its own data) off-host

• Protocol confirmation of receipt / detection of failure by subordinate chalk processes (leader died?)

Result

Testing:

• 10 chalk instances, all configured for heartbeat every 5 seconds, one should be able to report on behalf of itself and all other wanting to heartbeat. We should receive all of the heartbeats in 1 connection every 5 seconds

[viega]

I don't think they need to share a config; only one should be allowed to touch the local file system for non-tmp files though. Like any output logs, the report cache, etc.