search

terascope / teraslice

Scalable data processing pipelines in JavaScript
https://terascope.github.io/teraslice/
Apache License 2.0
50 stars 13 forks source link

Dynamic tracing or logging #754

Open erik-stephens opened 6 years ago

erik-stephens commented 6 years ago

As an operator, would be nice to open & close the log flood gates on demand, specifically the librdkafka logs. Some options:

  1. The bunyan logger has support for dtrace but only on systems with good dtrace support (linux not being one of them).

  2. Linux is slowly catching up to dtrace-like functionality. There is growing support for userspace dynamic tracing (USDT). Here is an example that shows how to define userspace probes in node & python.

  3. Roll our own, even if specific to getting at librdkafka debug logs. Spit-balling a workflow:

    1. Operator connects to some control channel nc ts_gen1 5000 <<< "probe job ${JOB_ID}"
    2. Start listening to consumer's event.log & event.error events
    3. The event handler relays all msgs to the probe channel
    4. When operator closes socket, remove that event listener

  4. Put log msgs into a ring buffer and expose an api for operators to dump the buffer. May still call for a way to dynamically enable the ring buffer since volume of msgs could introduce an unnacceptable time penalty. And will probably only want to see msgs for a specific job/consumer.

Option 1 is a no-go. I think option 2 worth exploring even if the USDT landscape is still a bit difficult to navigate in linux. I see options 3 & 4 as fallbacks.

kstaken commented 5 years ago

This is will be related to #846