Displaying meta-data in TensorBoard

[purpledog]

Hi,

I'd like to display information regarding the experiment I am running. For instance, I'd like to display the hyper-parameters distribution I used. This could be a small paragraph but a single line would do as well. Could this be dumped in the log directory so that TensorBoard pick it up and display it somewhere on the page? Or maybe passed as command-line flags?

Thanks, Frank

[wchargin]

Hi Frank,

A common practice is to use one run per assignment of hyperparameters, and encode the hyperparameters in the name of the run. For example, if the hyperparameters are the learning rate and the number of fully connected layers, runs could be of the form train_lr=1e-3_fc=5. In this case, the information will be displayed nicely in the runs selector and in the visualizations themselves. (This also has the benefit that you can filter by runs with specific hyperparameters, like train.*fc=[3-5].)

Would this work for you, or is something missing?

[teamdandelion]

Another approach would be to build a plugin (aka a dedicated dashboard) for displaying hyperparameters.

[purpledog]

William, the "name of the run" is the directory name right? Unfortunately, we use this for something else already. Also note that I want to display some text describing the distribution over all the runs. Typically something like:

"100 runs sweeping over the parameters:

• learning rate in [0.01, 0.0001]
• epsilon in [1e-7, 1e-8]"

Dan, I don't know much about TensorBoard plugins but note that this is not per-run data. As described above, the "meta data" I'd like to display is describing a set of runs. For instance, the file could be stored in the parent directory.

my_tb_log/tensorboard_metadata.json my_tb_log/run_0/ my_tb_log/run_1/ ... my_tb_log/run_99/

[wchargin]

Ah, I see. Yes, that's correct: it's the directory name.

Given that you want cross-run data, this isn't something that any of our existing plugins handle—but the infrastructure should be sufficient to create the plugin that you want. Here's how I'd suggest structuring it: on each run, emit a (single) summary containing the run's hyperparameters.* Then, create a plugin that reads this summary datum from each run in which it exists, and aggregates it however you want. (The simplest way to do this would be to just throw all the hyperparameter values into a list.) Does this makes sense?

• I suppose that you could even stash these in the graph, which might be good for a proof-of-concept but eventually sub-optimal.

If you just want the ability to manually add a paragraph that's displayed somewhere in the UI, you could fork the text plugin to make some "deduplicating message plugin," such that when the message is constant across all runs you only see one copy of it.

[purpledog]

What about an additional command like argument to the TensorBoard binary, something along the line of:

--title="one liner describing my experiment"

??

[teamdandelion]

I'm with William on preferring having an "experiment plugin" or "overview plugin" where lots of top-level information about the contained experiments is stored, maybe including titles, long readable descriptions, links, etc.

IMO, having an additional command line argument would be a mistake. It would push the responsibility to the user to figure out, every time they turn on tensorboard, which data is stored in the experiments they are pointing to, and it would be easy to get it wrong and mismatch titles and datasets. The metadata should be stored with the data, not remembered manually.

[ahowe42]

I second both the desire for a better way to display both run-specific and cross-run hyperparameters / metadata in tensorboard. Ideally, there would be a set of plots showing a target dev st evaluation metric wrt each hyperparameter to facilitate model comparison.