Optimised restyle/relayout, richer single-point manipulation/on-hover API

This is part feature request, part frustration, part a plea for help!

A bit of background:

(I am using plotly R for this project, but the core of the issue is in plotly.js) I have a 10000+ point scatter-geo plot and each point has multiple attributes. Creating a text or hovertext parameter is all well and good, but the resultant json object is massive (in excess of 100MB!!!). A vast majority of this is the repeated static text within the text/hovertext parameter, eg:

<b>Field 1</b>: {value1}<br>
<b>Field 2</b>: {value2}<br>
<b>Field 3</b>: {value3}<br>
<b>Field 4</b>: {value4}<br>
<b>Field 5</b>: {value5}

Yes, I could use hovertemplate and customdata, (except there might currently be a bug around frame column + customdata length validation in plotly-R preventing this <<insert bug # if issue created>>). In my case, most of the point specific data displayed in the hover tooltip is actually a result of a lookup. With that in mind, I set off to try and find a way of doing the lookup mapping client side using some JS callbacks to modify the hover tooltip text (plotly_hover for example) thinking it would be trivial....

Background reading:

I started looking at the plotly_beforehover and plotly_hover events as documented: https://plotly.com/javascript/hover-events/ however I quickly realised that some of the methods used within the documentation refer to un-documented modules/functions (specifically Plotly.Fx.hover in https://plotly.com/javascript/hover-events/#capturing-hover-events-data) and further googling led me to believe that they were intentionally exposed: https://github.com/plotly/plotly.js/issues/1963#issuecomment-378038797. There may be some low-level/undocumented functions that do what I need, but as of yet, I don't know what they are.

I noticed that the plotly_beforehover event can be used to stop the default hover behaviour, however it doesn't allow me to modify the event data prior to the hover event processing (I did manage to using Plotly.restyle, see next para). Unfortunate, because this would have been the ideal place to do it I believe as the plotly_hover event appears to have generated the tooltip before firing(?). With the currently exposed d3 api (which is to be removed in v2) I believe I could do what I need here by modifying the internal data structures prior to the rest of the hover function doing it's processing. This isn't a long term solution as when v2 of plotly.js comes out, access to the d3 api will no longer be available.

I then stumbled across a comment in a previous issue: https://github.com/plotly/plotly.js/issues/144#issuecomment-276190947 which highlights how a single marker can be modified using Plotly.restyle. Sadly, whilst this worked for me, it was horrendously slow and further issue browsing led me to the following: https://github.com/plotly/plotly.js/issues/1847 and specifically one of the comments: https://github.com/plotly/plotly.js/issues/1847#issuecomment-312927545

With the current `plotly.js` API

One option is to simply call Plotly.restyle and/or Plotly.relayout - which has the following issues:

1. These APIs weren't desinged for high FPS interactions, each call can trigger a _lot_ of recalculations, whether they're needed or not

2. Often, the desired effect needs _multiple_ `restyle` and `relayout` calls; currently can't be batched

3. The alternative, e.g. `Plotly.newPlot` pretty much throws out everything so it's not fast either, and resets user selections or other user navigated state (zoom, pan etc.)

4. The `plotly.js`-style column array updates are great when only one thing changes, but (I think) it's not possible to update one or a few points out of eg. 1000 data points, and even if we make such an API, making it faster than a full vector update is still a challenge.

5. The internal architecture isn't (yet?) reactive, i.e. while the above constraints can be mitigated eg. by batching, the real solution would be the principle of not recalculating things that have no need to change, which is an operational benefit of reactive data flows (most.js, kefir, flyd, vega-dataflow, or our compact crosslink library)

This leads me to believe that what I want to do, might actually not be possible at all with the current API/performance issues.

Final words: (TLDR)

With large datasets plotly's official Plotly.restyle API is just too slow to update single markers, (changing the text or color value of a marker on hover for example).

The current (undocumented) Plotly.Fx.hover API can only be used to manually trigger a hover event, but not manipulate the data content/display content. When used as per the example, we already know the points being hovered over so why can't we provide some text/display content?

Using plotly_beforehover to prevent the default tooltip being used requires that you implement your own unhover event+processing somehow, better to set the opacity of the default tooltip to 0%.

There appears to be no way of manipulating the tooltip generated by plotly without modifying the hovertemplate, text or hoverinfo parameters (which depends on Plotly.restyle....)

plotly / plotly.js