search

divviup / janus

Experimental implementation of the Distributed Aggregation Protocol (DAP) specification.
Mozilla Public License 2.0
52 stars 14 forks source link

Adopt `tokio-rs/console` #17

Open tgeoghegan opened 2 years ago

tgeoghegan commented 2 years ago

tokio-rs/console allows real-time insights into async tasks in a Tokio runtime. This would be an immensely powerful debugging and observability tool. We should integrate this into janus_server, and then figure out how to access this information in running instances in GKE.

divergentdave commented 2 years ago

To kick the tires on tokio-console, I added console-subscriber and enabled the required runtime traces on prio-server, see this commit. https://github.com/divergentdave/prio-server/commit/3543e89d86aed54714ebadc4262067bf53ad314c. Lessons learned: the difference between per-layer filters and global filters in tracing is very important. We must have at least one global filter enable the tokio-internal trace statements it relies on, and we can't have any global filters turn them off. (Note that EnvFilter only acts as a global filter, as it only implements Layer, while Targets may be used as a per-layer filter or as a layer itself, in which case it acts as a global filter)

The subscriber and tokio-console communicate via TCP port 6669 by default, so it should be straightforward to use this remotely via kubectl port-forward.

My main hesitation over turning this on is regarding the implications of RUSTFLAGS="--cfg tokio_unstable" -- what other behavior changes could we be opting in to here, and how much disruption could this introduce with future version upgrades?

branlwyd commented 2 years ago

tokio_unstable makes me a nervous, too. Per the tokio docs, tokio_unstable means:

The public API may break in 1.x releases.

This is unfortunate, in the sense that it means that after a Tokio update we might be stuck not being able to upgrade until one of our dependencies fixes a backwards-compatibility issue of their own.

(edit: of course, this applies to many, many Rust libraries at the moment, so that doesn't necessarily make this a dealbreaker...)

divergentdave commented 2 years ago

Subscriber support for this landed in #45. Is there anything else we want to do regarding the network transport aside from kubectl port-forward?