Show only plots used by currently selected runs

[danijar]

Right now, all plots used by any of the runs in the observed directory are shown. If the currently selected runs use just a sub set, the remaining plots are empty yet displayed. This means the user has to scroll through many empty plots to find the used ones.

[wchargin]

If the currently selected runs use just a sub set, the remaining plots are empty yet displayed.

Correct.

It wouldn't be too hard to filter the tags to only include those with selected runs. But this would have some unexpected consequences, mostly due to the limitations of Polymer's data binding. For instance, changing which runs are selected would often cause every chart to reload all its data, and would also make individual chart settings (like the vertical scale type and the current plot window) appear to shift across charts in undesirable ways.

How much would a search feature help your use case? (Prototype image.)

[danijar]

I don't see how the search feature helps here. I never used Polymer but it seems like hiding unused charts rather than removing their DOM tree could be a solution, no?

[wchargin]

If you're trying to "find the used ones", then you could search for them.

Yes, one would certainly expect that Polymer should make it possible to hide unused charts rather than removing their DOM tree. But it doesn't. The problem is that Polymer has no way to say, "this chart is associated with this tag". If the set of visible tags changes from [1, 2, 3, 4, 5] to [2, 3, 4, 5], then instead of simply removing or hiding Chart 1, Polymer will say, "okay: Chart 1, load Chart 2's data; Chart 2, load Chart 3's data, …; Chart 4, load Chart 5's data; remove Chart 5 from the DOM."

The concept that we want is very simple: give each child a key that is used by the reconciler. The Polymer team have told me that there is no way to do this in Polymer 1.x, and that I should file a feature request for 2.x…but that won't help us, as we are not planning to migrate to 2.x.

[teamdandelion]

(Sidebar: We probably will be forced to migrate to Polymer 2 eventually.)

[danijar]

Each plot should know the curves it's displaying, right? So it should know if it has no curves to display. Then it could activate a CSS rule to hide itself. Again, I'm not familiar with Polymer but it seems like Polymer shouldn't interact with this unless it doesn't allow components to see their children or change their own CSS.

[wchargin]

In that case, you would have the following effects: (a) the "number of charts" displayed in the header of each category would be incorrect; (b) pagination would be totally broken, as each page would display 12 chart components but anywhere from 0 to 12 actually visible charts. In general, it breaks the invariant that <tf-scalar-chart> displays a scalar chart, which isn't really something that we want to do.

@dandelionmane and I chatted briefly about this. We're not opposed to the idea, but it's not high-priority for us now, and it would take a good deal of fiddling to get right. PRs welcome; feel free to reopen.

[danijar]

I see. Shouldn't we keep this issue open since the problem is not solved? It's still an open bug even though the priority might be low.

[danijar]

@chihuahua Picking up the discussion from the closed duplicate bug #351:

If so, I wonder how we can design this feature so that charts don't shift too much when the user toggles runs.

This may not be necessary for now. The main problem is that when changing the graph code, summaries often get different scope prefixes and the result is having most of the displayed summaries empty. Talking to a few people, it seems like I'm not the only one slowed down by this.

@wchargin You suggested searching for the non-empty summaries. That doesn't work if there are many copies of the same summary with different scope prefixes, all but one being empty. Unfortunately, one often cannot afford re-running previous experiments when summary names changed so that the summaries match up nicely.

What would be the easiest approach to hide summaries without data? Ideally, hidden summaries would also contribute to a faster UI but even just hiding them would be a win in usability.

[rkaplan]

This would be a really helpful feature—any updates here?

[danijar]

Last time we talked this seemed unfeasible because of restrictions in the Polymer framework or would at least require a major rewrite. This bugfix/feature would still save me and my collaborator time and hassle every day, but I think for now it's not going to happen. Please correct me if I'm wrong, @wchargin :)

[wchargin]

Last time we talked this seemed unfeasible because of restrictions in the Polymer framework or would at least require a major rewrite. This bugfix/feature would still save me and my collaborator time and hassle every day, but I think for now it's not going to happen. Please correct me if I'm wrong, @wchargin :)

Unfortunately, you are correct. Related is #235, which is also a feature that we’d like but are unable to implement due to Polymer restrictions. I’m sorry that I don’t have a better answer for you.

[nfelt]

I found #55 which was filed earlier for the same issue as this one (as a migration of https://github.com/tensorflow/tensorflow/issues/7720), but I opted to dedupe that one in favor of this one since it has more context.

[phisad]

For me this regex workaround worked:

Tags matching val_acc|val_loss|^(?!class1|class2|class3).*$

Would be nice to have a switch for the tag matching from whitelist to blacklist. Then the user can "simply" put the charts she doesnt want to see.

[danijar]

I don't think that a regex solves the problem, but it does make me wonder why this is so difficult given that the search panel already supports dynamically choosing summary tags and updating pagination correctly.

[nfelt]

@danijar that's a good question - there are a number of difference that make handling the run selection case harder.

Re: the bad behavior that @wchargin was concerned about - the regex search actually does exhibit this today. For example, vertical scales and whether the chart is expanded or not stick with the actual tf-scalar-card element even when it's reassigned to display a new tag, which is a bug. Also, the charts do needlessly reload their data every time they shift around. Both of these issues are easier to overlook when confined to just the regex search results, but bringing them to all the normal scalar charts would make them unavoidable and significantly more painful - especially the chart-reloading issue.

That said, @stephanwlee has some ideas for how we can address this problem and we are hoping to make progress on it soon.

Re: pagination - updating it correctly for the regex search is less of a concern because users entering a new search query expect that the set of tags/charts displayed will change. So preserving the "semantic page" that they're on is less important. In contrast, users might update the selected runs with the intention of continuing to look at the exact same set of charts in front of them (just with different runs displayed on the charts). If changing the run selection affects the display of charts on page 1, then the "ripple effect" of shifting all the charts around on subsequent pages is more disruptive. Now, maybe that's just what we decide to live with, but I think it is a more complicated decision to do the right thing for this case.

[danijar]

Thanks for the insights and great to hear that Stephan is looking into it. Regarding your last paragraph, I don't think this would be much of an issue. If somebody wanted to compare runs, they would most likely select them at the same time. If you change the selection to a different experiment with different summary tags, I think it is reasonable that the page layout changes.

[timlod]

Is there possibly any progress around this issue? I'm logging many experiments with different metrics, and there are pages upon pages of metrics to sift through to find the ones actually containing data.

[bileschi]

Thanks @timlod for raising awareness to this longstanding issue. There has not been further work in this direction, but I highlighted this issue for UX exploration, possibly adding an affordance to switch between showing / hiding empty charts.

[TV4Fun]

Failing selecting metrics automatically, could we just have a way to manually exclude metrics we don't want cluttering our summary? All of the metrics I am currently using would fit on one page, but the extra unused metrics means I have to scroll through multiple pages to find the data I need.

[WhymustIhaveaname]

This would be a really helpful feature. I really need this feature.

[Jasonnor]

Hi, I wrote a simple chrome extension a few years ago to hide empty panes/scalars and it still seems to work.

https://github.com/Jasonnor/tensorboard-empty-scalar-hider

Hope it helps! 😃