Plot options with datashaded VectorField

[kcpevey]

The magic for VectorField (from the user guide):
%%opts VectorField [rescale_lengths=True size_index='Magnitude' color_index='Magnitude'] (scale=0.005)

This works for regular plots, but has no effect when I attempt to datashade the VectorField. How would I go about achieving the same effect, but with a datashaded plot?

I tried switching to the not-yet-released hv.util.opts but that didn't appear to have much of an effect (only colored vectors all black).

I also tried applying the parameters to the datashade() call, but it wouldn't take those parameters.

[philippjfr]

This works for regular plots, but has no effect when I attempt to datashade the VectorField. How would I go about achieving the same effect, but with a datashaded plot?

Datashader does not know about or understand a VectorField and I suspect this goes way beyond what datashader will ever support so I'm afraid there isn't anything you can do here.

[kcpevey]

So its not possible to datashade a VectorField in any useful way (i.e. one where I have some control over display opts?)

This quote

Added VectorField element and datashading operation to plot small and large quiver plots and other collections of vectors (#122)

From these release notes made me think it was possible. http://blog.holoviews.org/release_1.5.html

[kcpevey]

My dataset is WAY to big to plot in a regular VectorField. I have to find some way around that.

[philippjfr]

Added VectorField element and datashading operation to plot small and large quiver plots and other collections of vectors (#122)

That is indeed an error, I think it should say project operation.

[philippjfr]

My dataset is WAY to big to plot in a regular VectorField. I have to find some way around that.

We could probably figure out a way to convert the VectorField to a list of paths, which could be datashaded.

[kcpevey]

Ideas for a workaround?

[kcpevey]

Oh boy that sounds... fun? Let's think on this and discuss on Thursday.

[philippjfr]

Actually turned out to be reasonably straightforward, here's a first cut:

import param
from holoviews import Operation
from holoviews.core.util import basestring
from holoviews.util.transform import dim
from holoviews.plotting.util import get_min_distance

class vectorfield_to_paths(Operation):
    arrow_heads = param.Boolean(default=True, doc="""
        Whether or not to draw arrow heads.""")

    color = param.ClassSelector(class_=(basestring, dim, int), default=3)

    magnitude = param.ClassSelector(class_=(basestring, dim), doc="""
        Dimension or dimension value transform that declares the magnitude
        of each vector. Magnitude is expected to be scaled between 0-1,
        by default the magnitudes are rescaled relative to the minimum
        distance between vectors, this can be disabled with the
        rescale_lengths option.""")

    rescale_lengths = param.Boolean(default=True, doc="""
        Whether the lengths will be rescaled to take into account the
        smallest non-zero distance between two vectors.""")

    pivot = param.ObjectSelector(default='mid', objects=['mid', 'tip', 'tail'],
                                 doc="""
        The point around which the arrows should pivot valid options
        include 'mid', 'tip' and 'tail'.""")

    scale = param.Number(default=1)

    def _get_lengths(self, element):
        mag_dim = self.magnitude
        if isinstance(mag_dim, basestring):
            mag_dim = element.get_dimension(mag_dim)

        (x0, x1), (y0, y1) = (element.range(i) for i in range(2))
        if mag_dim:
            if isinstance(mag_dim, dim):
                magnitudes = mag_dim.apply(element, flat=True)
            else:
                magnitudes = element.dimension_values(mag_dim)
            if self.p.rescale_lengths:
                base_dist = get_min_distance(element)
                magnitudes *= base_dist
        else:
            magnitudes = np.ones(len(element))
            if self.rescale_lengths:
                base_dist = get_min_distance(element)
                magnitudes *= base_dist

        return magnitudes

    def _process(self, element, key=None):
        # Compute segments and arrowheads
        xs = element.dimension_values(0)
        ys = element.dimension_values(1)

        rads = element.dimension_values(2)

        lens = self._get_lengths(element)/self.p.scale

        # Compute offset depending on pivot option
        xoffsets = np.cos(rads)*lens/2.
        yoffsets = np.sin(rads)*lens/2.
        if self.pivot == 'mid':
            nxoff, pxoff = xoffsets, xoffsets
            nyoff, pyoff = yoffsets, yoffsets
        elif self.pivot == 'tip':
            nxoff, pxoff = 0, xoffsets*2
            nyoff, pyoff = 0, yoffsets*2
        elif self.pivot == 'tail':
            nxoff, pxoff = xoffsets*2, 0
            nyoff, pyoff = yoffsets*2, 0
        x0s, x1s = (xs + nxoff, xs - pxoff)
        y0s, y1s = (ys + nyoff, ys - pyoff)

        if self.arrow_heads:
            arrow_len = (lens/4.)
            xa1s = x0s - np.cos(rads+np.pi/4)*arrow_len
            ya1s = y0s - np.sin(rads+np.pi/4)*arrow_len
            xa2s = x0s - np.cos(rads-np.pi/4)*arrow_len
            ya2s = y0s - np.sin(rads-np.pi/4)*arrow_len

            xs = [[x0, x1, np.nan, x0, xa1, np.nan, x0, xa2]
                  for x0, x1, xa1, xa2 in zip(x0s, x1s, xa1s, xa2s)]
            ys = [[y0, y1, np.nan, y0, ya1, np.nan, y0, ya2] 
                  for y0, y1, ya1, ya2 in zip(y0s, y1s, ya1s, ya2s)]
        else:
            xs = [[x0, x1, np.nan] for x0, x1 in zip(x0s, x1s)]
            ys = [[y0, y1, np.nan] for y0, y1 in zip(y0s, y1s)]

        if isinstance(self.p.color, dim):
            colors = self.p.color.apply(element, flat=True)
        else:
            colors = element.dimension_values(self.p.color)

        return hv.Path([{'x': x, 'y': y, 'color': c} for x, y, c in zip(xs, ys, colors)], vdims='color')

x,y  = np.mgrid[-10:10,-10:10] * 0.25
sine_rings  = np.sin(x**2+y**2)*np.pi+np.pi
exp_falloff = 1/np.exp((x**2+y**2)/8)

vector_data = (x,y,sine_rings, exp_falloff)
vf = hv.VectorField(vector_data)

datashade(vectorfield_to_paths(vf), aggregator='mean', precompute=True)

[jbednar]

And is it reasonably responsive, or at least more than for a regular vector field?

[philippjfr]

As long as you precompute, i.e. cache the nan concatenated paths.

[philippjfr]

Note that the rescale_lengths option slows things down a lot both when used as a plot option and in the operation above. The alternative is to provide a scale yourself by experimenting a bit, although using datashader drawing large, overlapping arrows can be quite pretty:

[philippjfr]

@kcpevey Here's an updated version which is about 20x faster and much less memory hungry than my initial approach above. I've removed the rescale_lengths option entirely since it's never going to be useful for large vectorfields.

class vectorfield_to_paths(Operation):

    arrow_heads = param.Boolean(default=True, doc="""
        Whether or not to draw arrow heads.""")

    color = param.ClassSelector(class_=(basestring, dim, int), default=3)

    magnitude = param.ClassSelector(class_=(basestring, dim, int), default=3, doc="""
        Dimension or dimension value transform that declares the magnitude
        of each vector. Magnitude is expected to be scaled between 0-1,
        by default the magnitudes are rescaled relative to the minimum
        distance between vectors, this can be disabled with the
        rescale_lengths option.""")

    pivot = param.ObjectSelector(default='mid', objects=['mid', 'tip', 'tail'],
                                 doc="""
        The point around which the arrows should pivot valid options
        include 'mid', 'tip' and 'tail'.""")

    scale = param.Number(default=1)

    def _get_lengths(self, element):
        mag_dim = self.magnitude
        if mag_dim:
            if isinstance(mag_dim, dim):
                magnitudes = mag_dim.apply(element, flat=True)
            else:
                magnitudes = element.dimension_values(mag_dim)
        else:
            magnitudes = np.ones(len(element))
        return magnitudes

    def _process(self, element, key=None):
        # Compute segments and arrowheads
        xd, yd = element.kdims
        xs = element.dimension_values(0)
        ys = element.dimension_values(1)

        rads = element.dimension_values(2)

        lens = self._get_lengths(element)/self.p.scale

        # Compute offset depending on pivot option
        xoffsets = np.cos(rads)*lens/2.
        yoffsets = np.sin(rads)*lens/2.
        if self.pivot == 'mid':
            nxoff, pxoff = xoffsets, xoffsets
            nyoff, pyoff = yoffsets, yoffsets
        elif self.pivot == 'tip':
            nxoff, pxoff = 0, xoffsets*2
            nyoff, pyoff = 0, yoffsets*2
        elif self.pivot == 'tail':
            nxoff, pxoff = xoffsets*2, 0
            nyoff, pyoff = yoffsets*2, 0
        x0s, x1s = (xs + nxoff, xs - pxoff)
        y0s, y1s = (ys + nyoff, ys - pyoff)

        if isinstance(self.p.color, dim):
            colors = self.p.color.apply(element, flat=True)
        elif self.p.color is not None:
            colors = element.dimension_values(self.p.color)
        else:
            colors = None

        if self.arrow_heads:
            arrow_len = (lens/4.)
            xa1s = x0s - np.cos(rads+np.pi/4)*arrow_len
            ya1s = y0s - np.sin(rads+np.pi/4)*arrow_len
            xa2s = x0s - np.cos(rads-np.pi/4)*arrow_len
            ya2s = y0s - np.sin(rads-np.pi/4)*arrow_len
            xs = np.empty((len(x0s)*9))
            ys = np.empty((len(x0s)*9))
            cvals = np.empty((len(x0s)*9))
            for i, (x0, x1, xa1, xa2, y0, y1, ya1, ya2) in enumerate(zip(x0s, x1s, xa1s, xa2s, y0s, y1s, ya1s, ya2s)):
                slc = slice(i*9, i*9+9)
                xs[slc] = [x0, x1, np.nan, x0, xa1, np.nan, x0, xa2, np.nan]
                ys[slc] = [y0, y1, np.nan, y0, ya1, np.nan, y0, ya2, np.nan]
                if colors is not None:
                    cvals[slc] = colors[i]
        else:
            xs = np.empty((len(x0s)*3))
            ys = np.empty((len(x0s)*3))
            for x0, x1, y0, y1 in enumerate(zip(x0s, x1s, y0s, y1s)):
                slc = slice(i*3, i*3+3)
                xs[slc] = [x0, x1, np.nan]
                ys[slc] = [y0, y1, np.nan]
                if colors is not None:
                    cvals[slc] = colors[i]

        if colors is None:
            data = [(xs, ys)]
            vdims = []
        else:
            data = [(xs, ys, cvals)]
            vdims = [str(self.p.color)]

        return element.clone(data, vdims=vdims, new_type=hv.Path, datatype=['multitabular'])

Here's a 500x500 VectorField datashaded in about 12 seconds:

[kcpevey]

Much much improved! I can now render my 836k vector dataset. Initial draw time is not terrible, but I'm using this function for a dynamic map (time series). So every time I change the time, I'm waiting ~4 seconds for it to rerender. I tried precompute=True inside the datashade call, but that didn't seem to have an effect. I thought that would help with the lag when moving my time slider?

[philippjfr]

No, precompute caches the data per time step and in this case happens to be entirely unnecessary because the operation already flattens the data. Don't think there's anything we can do about that delay unfortunately.

[kcpevey]

Calling this one resolved.

[danwild]

This is incredibly useful - thanks @philippjfr !