RustBoard tasks

[wchargin]

Feature tasks

If you’re interested, let me know, and I’ll assess whether it’s a good idea and provide guidance if appropriate.

Small-ish

• [x] @bileschi: Plugin listing (#4487)
  • touches: server
  • straightforward; good starter task
• [x] @wchargin: Data location and experiment metadata (#4751)
  • touches: server, Python data provider, RPC protos
  • pretty straightforward; good starter task
• [x] @wchargin: Make CRC checking optional (#4573)
  • touches: core, CLI
  • straightforward; good starter task
• [x] @nfelt: Online commits (commit data periodically while runs are loading) (#4604)
  • touches: core only
  • fairly straightforward; good starter task
• [x] @psybuzz: Samples-per-plugin (#4689)
  • touches: core, CLI, Python ingester
  • probably pretty straightforward? could be a good starter task
  • [x] optional extra: support unbounded reservoirs (#4754)

Involved

• [x] @nfelt: Tensors core (#4726)
  • touches: core, server, Python data provider
• [x] @nfelt: Tensors data compat
  • [x] histograms (#4740)
  • [x] PR curves (#4734)
  • [x] hparams (#4735)
  • [x] text (#4734)
• [x] @wchargin: Blob sequences (#4563, et al.)
  • touches: core, server, Python data provider
  • needs generic blob sequence support plus compat for images, audio, graphs
  • expect some moderately tricky stuff on the server since ReadBlob is a server-streaming RPC
• [x] @wchargin: Blob data compat
  • [x] images (#4564)
  • [x] audio (#4566)
  • [x] graphs (#4544)
  • [x] graphs sub-plugins (Keras, etc.) (#4568, #4569)
• [x] @wchargin: Parallelize across runs (#4634)
  • touches: core only
  • expect to think about lifetimes
  • consider ~crossbeam for scoped threads and maybe SegQueue~ rayon!
• [ ] Optimize run-tag filters over small input sets
  • i.e.: iterate over the filter, not the map
  • touches: server only
  • will be harder than it sounds :-)
  • mostly for learning purposes; will be faster, but won’t matter in practice
• [ ] Reload-multifile
  • touches: core, CLI
  • note: current logic does always read from all files; it just never marks old files as dead

These are mostly for feature parity. There are further ideas for new features—indexing, deferred blob loading, etc.—but let’s hold off on those.

Infra tasks

(I plan to take care of these.)

• [x] Filesystems (#4636)
  • touches: core, CLI
• [x] Pull main.rs into rustboard_core::cli
  • dovetails with filesystems: bring your own entry point
• [x] Pip packaging
  • [x] set up package structure and integrate with TensorBoard
  • [x] release Pip package (pre-release is okay)
  • [x] add Pip dependency from tensorboard to tensorboard-data-server (#4676)
• [x] Cross-platform builds
  • trivial to cross-compile to pc-windows-gnu, less so to pc-windows-msvc
  • osxcross?
  • just build ’em all on GitHub Actions? has VMs for all relevant platforms
• [x] Benchmarks
  • touches: core, build
  • status: have a starting point; can expand as needed

Other tasks that I plan to investigate:

• [x] Graph plugin parity (#4473, et al.)
  • touches: Python graph plugin, both data_compat
  • profile data, non-run graphs with data provider
• [x] Projector plugin parity (#4494, et al.)
  • touches: Python projector plugin
  • just needs to work without multiplexer; can still hit local disk rather than actually using the data provider
• [x] Mesh plugin parity (#4775)
  • touches: Python mesh plugin
  • just needs to rewire multiplexer to data provider, using tensors (see discussion on #4734)
• [x] Custom scalars plugin parity (#4774)
  • touches: Python custom scalars plugin, one-line changes to both datacompats
• [x] DebuggerV2 well-behavedness (#4659)
  • needs to not hammer GCS
  • not strictly RustBoard, but stops --load_fast from being fast

[bileschi]

May I reserve plugin listing?

[wchargin]

Sure! You’ll want to update the list_plugins RPC handler to iterate over the map of data and extract the plugin names from the summary metadata, rather than just returning a hard-coded "scalars".

You can unit test this easily (follow existing examples), and for an end-to-end test you can compare a logdir with scalars to an empty logdir (since we don’t yet have tensor/blob sequence support).

[wchargin]

I’ve grabbed blob sequence support. It’s more involved than tensors but teaches approx. the same lessons, so I figure that I can grab it.

[wchargin]

I ended up also grabbing “make CRCs optional” because I was curious and it turns out to make things 15–60% faster (#4573), so I didn’t want to just leave that on the table that it might potentially get dropped.

[wchargin]

Thanks to @bileschi, @psybuzz, and @nfelt for your contributions! We’ve gotten all the important stuff by this point, so I’m closing this.