Upgrading to v1.14: "TypeError: dashboardsTemplate is null"

ceevee830 commented 5 years ago

Hi William,

Here is that console output I spoke about at our meeting last week. We're upgrading to v1.14 and are coming across this error. Nothing appears in the browser (ie, no toolbar, no navigator, no orange). It sounds like I need to register the dashboard. Sorry if I missed the announcement, but time is of the essence. Thanks,

-Chris

goog.require could not find: jspb.utils
Error: goog.require could not find: jspb.utils
goog.require could not find: jspb.utils
Error: goog.require could not find: jspb.Message
goog.require could not find: jspb.Message
SyntaxError: export declarations may only appear at top level of a module
SyntaxError: export declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
SyntaxError: export declarations may only appear at top level of a module
SyntaxError: export declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
SyntaxError: export declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
SyntaxError: export declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
SyntaxError: export declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
SyntaxError: export declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
SyntaxError: import declarations may only appear at top level of a module
TypeError: dashboardsTemplate is null
    ready http:/
    _readySelf http:/
    _ready http:/
    _tryReady http:/
    _initFeatures http:/
    __initialize http:/
    createdCallback http:/
    i http:/
    n http:/
    t http:/
    n http:/
    o http:/
    t http:/
    n http:/
    n http:/
    n http:/
    t http:/
    o http:/
    t http:/
    v http:/
    i http:/
    r http:/
    w http:/
    t http:/
    <anonymous> http:/
    n http:/
    i http:/
    t http:/
    o http:/
    r http:/

wchargin commented 5 years ago

Hi Chris—

Under what conditions do you see that error?
Is this the stock tensorboard binary? or a custom build of TensorBoard from master or the 1.14 branch? or a custom build of TensorBoard with additional plugins?
If you remove any additional plugins, does the error persist?
What browser and what browser version are you using?
If this is the error that you mentioned at our meeting last week, did it also occur prior to v1.14?
In this build of TensorBoard, did you remove the compile = True build attribute?
Have you made any changes to the interactive_inference plugin (What-If Tool) or its build files?

Obviously it would be best if you could point us to the full source tree that generates this error; if it involves closed-source plugins, it would be great if you could strip those down to a minimal example that still has the problem and that you can share with us.

ceevee830 commented 5 years ago

Hi William, Under what conditions do you see that error?: Starting the TB back-end server is fine; this error occurs when the browser client starts. Nothing appears; console output above. (what kind of build)? This is the standard TB v1.14 with our plugins inside (and additions to GRAPH plugin) browser version?: Firefox 67.0.2 If this is the error that you mentioned at our meeting last week, did it also occur prior to v1.14 No, whatever we have in TB v1.13 is working fine. (I do not see this error in v1.13). compile = true? Not a part - just using stock TV v1.14 setting interactive_inference? No

TB v1.14 out of the box works perfectly for me. It surely seems like this is caused by our plugin. It seems like all TB plugins are now required to register their dashboards. .....or perhaps the method of registering plugins has changed in v1.14.
Investigation results: In v1.13, at the bottom of tf_graph_dashboard.html there was code like this: tf_tensorboard.registerDashboard({ plugin: 'graphs', elementName: 'tf-graph-dashboard', // TODO(@chihuahua): Reconcile this setting with Health Pills. isReloadDisabled: true, });
So our gpu_summary in TB v1.13 mimicks that.
Now in TB v1.14, tf_graph_dashboard.html no longer has that code. That dashboard registration no longer exists. Could this be to root cause of the problem?

Our source code is not public, but hopefully I can include some key pieces. The TB v1.14 error message comes from components/tf_tensorboard so I'll start there:

1) BUILD: See tf_gpu_summary_dashboard below

    name = "default_plugins",
    srcs = ["default-plugins.html"],
    path = "/tf-tensorboard",
    visibility = ["//visibility:public"],
    deps = [
        "//tensorboard/plugins/audio/tf_audio_dashboard",
        "//tensorboard/plugins/beholder/tf_beholder_dashboard",
        "//tensorboard/plugins/custom_scalar/tf_custom_scalar_dashboard",
        "//tensorboard/plugins/debugger/tf_debugger_dashboard",
        "//tensorboard/plugins/distribution/tf_distribution_dashboard",
        "//tensorboard/plugins/graph/tf_graph_dashboard",
        "//tensorboard/plugins/gpu_summary/tf_gpu_summary_dashboard",     (<----- here)
        "//tensorboard/plugins/histogram/tf_histogram_dashboard",
        "//tensorboard/plugins/hparams/tf_hparams_dashboard",
        "//tensorboard/plugins/image/tf_image_dashboard",
        "//tensorboard/plugins/interactive_inference/tf_interactive_inference_dashboard",
        "//tensorboard/plugins/mesh/tf_mesh_dashboard",
        "//tensorboard/plugins/pr_curve/tf_pr_curve_dashboard",
        "//tensorboard/plugins/profile/tf_profile_dashboard",
        "//tensorboard/plugins/projector/vz_projector",
        "//tensorboard/plugins/scalar/tf_scalar_dashboard",
        "//tensorboard/plugins/text/tf_text_dashboard",
    ],
)

2) default-plugins.html: See tf-gpu-summary-dashboard.html below


<link rel="import" href="../tf-custom-scalar-dashboard/tf-custom-scalar-dashboard.html">
<link rel="import" href="../tf-image-dashboard/tf-image-dashboard.html">
<link rel="import" href="../tf-audio-dashboard/tf-audio-dashboard.html">
<link rel="import" href="../tf-debugger-dashboard/tf-debugger-dashboard.html">
<link rel="import" href="../tf-graph-dashboard/tf-graph-dashboard.html">
<link rel="import" href="../tf-gpu-summary-dashboard/tf-gpu-summary-dashboard.html">     (<----- here)
<link rel="import" href="../tf-distribution-dashboard/tf-distribution-dashboard.html">
<link rel="import" href="../tf-histogram-dashboard/tf-histogram-dashboard.html">
<link rel="import" href="../vz-projector/vz-projector-dashboard.html">
<link rel="import" href="../tf-text-dashboard/tf-text-dashboard.html">
<link rel="import" href="../tf-pr-curve-dashboard/tf-pr-curve-dashboard.html">
<link rel="import" href="../tf-profile-dashboard/tf-profile-dashboard.html">
<link rel="import" href="../tf-beholder-dashboard/tf-beholder-dashboard.html">
<link rel="import" href="../tf-interactive-inference-dashboard/tf-interactive-inference-dashboard.html">```

wchargin commented 5 years ago

That dashboard registration no longer exists. Could this be to root cause of the problem?

Not directly. That registration code is still valid, and exists for some plugins for legacy reasons (cf., e.g., tf-profile-dashboard.html). There is a new way of registering dashboards (define frontend_metadata on the Python plugin; see existing plugins for examples), but the old way is still supported.

Your default-plugins BUILD and HTML look fine. If you remove the runtime and build dependencies on your plugin, such that

bazel query 'somepath(//your_tensorboard_binary, //your_plugin/...)

returns no results (where //your_tensorboard_binary is the thing that you bazel run and //your_plugin is the package with all the code for your frontend), does the problem persist?

Looking again at that error message, it’s not clear to me how dashboardsTemplate might be null. Can you try bisecting your TensorBoard dependency to find the first TensorBoard commit that has the problem?

ceevee830 commented 5 years ago

OK, so if the old way is still supported, I'd like to continue down that path.
Working on the bazel query. Can you help me with the cmdline? somepath? bazel query 'somepath(//bazel-bin/tensorboard/tensorboard, //gpu_summary/...)' Here is the output:

INFO: Invocation ID: 1a4d08d7-a91a-45d5-adff-189e1f8b0097
ERROR: no targets found beneath 'gpu_summary'
Loading: 0 packages loaded

So I copied out plugin code under plugins/gpu_summary. It compiled fine. Runtime was fine too, of course I could not see the plugin since it was not registered inside components/tf_tensorboard. Then updated components/tf_tensorboard/BUILD and components/tf_tensorboard/default-plugins.html. The compile is fine, but the browser then emits the error dashboardsTemplate is null. The code surrounding this error message is We have to wait for our dashboard-containers to be stamped before we can do anything.
Is that a clue? I put debug statements inside this loop like this:

          for (const container of this.querySelectorAll('.dashboard-container')) {

console.log(container);

            dashboardContainersStamped[container.dataset.dashboard] = true;
          }

and saw 16 containers:

<div class="dashboard-container style-scope tf-tensorboard" data-dashboard="audio">
<div class="dashboard-container style-scope tf-tensorboard" data-dashboard="beholder">
<div class="dashboard-container style-scope tf-tensorboard" data-dashboard="custom_scalars">
<div class="dashboard-container style-scope tf-tensorboard" data-dashboard="debugger">
<div class="dashboard-container style-scope tf-tensorboard" data-dashboard="distributions">
<div class="dashboard-container style-scope tf-tensorboard" data-dashboard="graphs" data-selected="">
<div class="dashboard-container style-scope tf-tensorboard" data-dashboard="histograms">
<div class="dashboard-container style-scope tf-tensorboard" data-dashboard="hparams">
<div class="dashboard-container style-scope tf-tensorboard" data-dashboard="images">
<div class="dashboard-container style-scope tf-tensorboard" data-dashboard="mesh">
<div class="dashboard-container style-scope tf-tensorboard" data-dashboard="pr_curves">
<div class="dashboard-container style-scope tf-tensorboard" data-dashboard="profile">
<div class="dashboard-container style-scope tf-tensorboard" data-dashboard="projector">
<div class="dashboard-container style-scope tf-tensorboard" data-dashboard="scalars">
<div class="dashboard-container style-scope tf-tensorboard" data-dashboard="text">
<div class="dashboard-container style-scope tf-tensorboard" data-dashboard="whatif">

But when I update components/tf_tensorboard/BUILD and components/tf_tensorboard/default-plugins.html to add my plugin/container, the loop emits zero containers. Ergo the error dashboardsTemplate is null. Sounds like I need to decorate my plugin somehow. Thoughts?

ceevee830 commented 5 years ago

Hi William, we were going to release our version of TB v1.13 last Friday, but TB 1.14 was released that day. I'm tasked to upgrade to v1.14 ASAP. Any cycles you could give to this issue would be greatly appreciated.

wchargin commented 5 years ago

But when I update components/tf_tensorboard/BUILD and components/tf_tensorboard/default-plugins.html to add my plugin/container, the loop emits zero containers.

The fact that adding your plugin causes the loop to emit zero containers rather than simply failing to add your plugin sounds like something in your plugin is breaking the rest of TensorBoard. Combined with the SyntaxErrors that you mention in your original post, perhaps check whether your files are being compiled as you expect?

Recent changes to TensorBoard’s BUILD (#2267, #2308) have required updating the version of jscompiler that we use, to keep up with breaking changes in Bazel. Some of those caused errors in the What-If Tool (#2312); perhaps you’re encountering similar problems?

Can you help me with the cmdline? somepath?

Docs for somepath are here.

Any cycles you could give to this issue would be greatly appreciated.

There’s not much that I can offer, unfortunately. Other than bisecting the TensorBoard dep as I suggested above, you might try to add debugger; statements to tf-tensorboard to determine why dashboardsTemplate is suddenly now null. But I’m afraid that the answer isn’t obvious to me, either.

ceevee830 commented 5 years ago

HI William,

Seeing that the dashboard extensions and dashboard loaders have changed between versions, I copied the GRAPHS plugin code to where my plugin used to be, changed the names to match, and am slowly replacing the GRAPHS business logic with mine. This is so that the dashboard and loader signatures have the same signatures. Things were going good until I found this error message again (dashboardsTemplate is null). I've narrowed it down to one line:

<script src="../webcomponentsjs/webcomponents-lite.min.js"></script>

When I run without this line, I can see the framework. When I run with this line compiled, it yields that error message at run time. It happens with other imports as well but not all.

Importing webcomponents-lite.min.js used to work in TB v1.13, and I see other components including this file in v1.14.

I'm stuck.... I'm suspecting Bazel for some reason. I'm at version 0.22.0. Which version do you recommend?

(I looked at those issues above regarding jscompiler and they do not appear to be related. The compiler errors I listed are from TB v1.14 out of the box)

ceevee830 commented 5 years ago

Hi William,

I've removed all the code in this html file to make a degenerative case that demonstrates the problem:

<script src="../webcomponentsjs/webcomponents-lite.min.js"></script>

<dom-module id="tf-gpu-summary-board">
</dom-module>

<script>
Polymer({
  is: 'tf-gpu-summary-board',
});
</script>

When I remove (or comment-out) <script src="../webcomponentsjs/webcomponents-lite.min.js"></script>, TB appears in the browser.

Bazel 0.22.0 Firefox browser 67.0.4 (64-bit) windows Firefox browser 67.0.3 (64-bit) ubuntu

stack trace

TypeError: dashboardsTemplate is null
    ready
    _readySelf
    _ready
    _tryReady
    _initFeatures
    __initialize
    createdCallback

Thoughts?

wchargin commented 5 years ago

Great; thanks for the minimal repro. Loading Polymer or the web components polyfills multiple times is definitely known to cause problems. Can you just remove this line? (From a quick grep, it doesn’t look like any of the other plugins include it in the main frame; it’s only used by trace_viewer.html, which appears in an iframe.)

I’m not sure what changed from 1.13 to 1.14 to surface this.

ceevee830 commented 5 years ago

OK, thanks, I removed the line <script src="../webcomponentsjs/webcomponents-lite.min.js"></script> from all the source files and the dashboardsTemplate problem has stopped.

But one thing I just read in your recent message was Loading Polymer or the web components polyfills multiple times is definitely known to cause problems. Regarding the polyfill: I'm noticing that my panels are being replaced with #document-fragment. Is this related? Something appears to be stripping out my panels. Thoughts?

I also noticed that my DIV that displays the 'above' and 'below' scaffolding strings has been changed to include this class style-scope. What is that? Here is that code:

<div>Above tf-gpu-summary-board panel</div>
    <tf-gpu-summary-board id="gpusummaryboard"
        graph="[[_graph]]"
    ></tf-gpu-summary-board>
<div>Below tf-gpu-summary-board panel</div>

wchargin commented 5 years ago

The style-scopeing is done by Polymer, as a polyfill for scoped CSS: styles defined in <style> elements inside your <dom-module> only apply to components specified in that <dom-module>, so you can use short, readable selectors like .icon without worrying about conflicts with other components.

The #document-fragment is an artifact of Shadow DOM itself. It’s the… “shadow root”, I believe? I’m not sure; I never grokked all the pieces to this, either.

wchargin commented 5 years ago

And glad to hear that the problem has stopped!

ceevee830 commented 5 years ago

That's good to know. The bigger issue is referred-to be the hand-drawn curly brace with the arrow. Where did the html for those two panels go? Shall I start a new issue?

stephanwlee commented 5 years ago

But one thing I just read in your recent message was Loading Polymer or the web components polyfills multiple times is definitely known to cause problems. Regarding the polyfill: I'm noticing that my panels are being replaced with #document-fragment. Is this related? Something appears to be stripping out my panels. Thoughts? Where did the html for those two panels go?

This is a normal behavior. When you define a Polymer component, you specify what the DOM should look like in a template which by itself is not user visible element. Polymer, during the runtime, parses the template HTML to figure out its data bindings and hides the template content away from the user. It, after parsing it, does hold the reference to inner templates in the memory which gets used only when a Polymer component is created via document.createElement (or HTML).

ceevee830 commented 5 years ago

Hi Stephan,

So I copied the GRAPHS plugin dashboard, board, info, controls, etc to get the new signatures of the new TB v1.14 interface. Where in the GRAPHS plugin is document.createElement used? I cannot seem to find it. Also, my plugin did not need document.createElement before - what changed?

stephanwlee commented 5 years ago

Where in the GRAPHS plugin is document.createElement used? I cannot seem to find it. Also, my plugin did not need document.createElement before - what changed?

I was trying to describe the internals of Polymer one does not need to know about. I brought it up while trying to explain why the template was empty and why you were only seeing document-fragment inside.

tensorflow / tensorboard

Upgrading to v1.14: "TypeError: dashboardsTemplate is null" #2359