redim.range all

[pjvalla]

What is the "pythonic" way to redimension all subplots inside of a layout?

I am working with this layout:

:Layout .Overlay.I :Overlay .Plot.Correlation.I :RGB [sample,timing_est_norm] (R,G,B,A) .Plot.Correlation.II :RGB [sample,enable_sig] (R,G,B,A) .Plot.Correlation.III :RGB [sample,max_sig] (R,G,B,A) .NdOverlay.Max_Value :NdOverlay [Element] :Points [x,y] .Overlay.II :Overlay .Plot.Correlation.I :RGB [sample,timing_est] (R,G,B,A) .Plot.Correlation.II :RGB [sample,auto_corr] (R,G,B,A) .NdOverlay.Auto_Corr :NdOverlay [Element] :Points [x,y] .Time.Real :RGB [sample,real] (R,G,B,A) .Time.Imag :RGB [sample,imag] (R,G,B,A)

Notice all the x axes are labelled "sample". This produces the output below. The parameter Dec Rate controls the decimation setting for the input signal. Reducing this should increase the range of the x-axis, while increasing Dec Rate shrinks the range (I can compute the appropriate range). The current code simply uses the initial range and does not expand or shrink the range.

I have tried redim.range each of the plots independently, but I think it is unsuccessful since all of the plots are linked. Is there a way to go through and redimension all of the plots in the final layout?

[pjvalla]

So I can change all x axes on startup using this (I tested using values well beyond/less than the actual data range)

layout = layout.redim.range(Sample=xrange_tuple)
return layout

However, the axes remain the same as the initial layout on successive calls.

[philippjfr]

However, the axes remain the same as the initial layout on successive calls.

Is the xrange_tuple different each time and have you tried setting framewise=True as an option on the individual plots?

[pjvalla]

The xrange_tuple is different each time. framewise=True is set for the dynamicmap (inside the plot_params function).

Here is the full code.

I edited the original code to remove local dependencies. You should be able to run this.

# -*- coding: utf-8 -*-F
"""
Created on 2018-10-15T12:51:30.217Z

"""

import scipy as sp
import scipy.io as sio
import numpy as np
import pandas as pd
import ipdb as ipdb

from bokeh.io import curdoc, output_notebook
from datashader.bokeh_ext import InteractiveImage
import holoviews as hv
import datashader as ds
from holoviews.operation.datashader import datashade, aggregate, dynspread
from holoviews.operation import decimate
import panel as pp
import param

from datashader.colors import Sets1to3 # default datashade() and shade() color cycle
hv.extension('bokeh','matplotlib')

class Burst_Params(param.Parameterized):
    num_list = range(16, 2049)
    Block_Size = param.ObjectSelector(default=500, objects=num_list)
    num_list = range(1, 2049)
    Corr_Size = param.ObjectSelector(default=200, objects=num_list)
    num_list = [str(i) for i in range(2, 12)]
    Num_Blocks = param.ObjectSelector(default='2', objects=num_list)

class Dec_Params(param.Parameterized):
    num_list = range(1, 1025)
    Dec_Rate = param.ObjectSelector(default=42, objects=num_list)

class Burst_Detector(object):
    def __init__(self, tog_cnt=5):
        self.plot_height = 400
        self.plot_width = 750
        self.min_alpha = 120
        self.on_thresh = .7
        self.off_thresh = .5
        self.df = None
        # time plot counts.
        self.time_yrange0 = None
        self.time_yrange1 = None

        self.block_size = 32
        self.corr_size = 32
        self.num_blocks = 1
        self.dec_rate = 1
        # self.dec_rate_time = 1
        self.update_rngs = True
        self.xrange_new = None
        self.yrange_new = None
        self.est_yrange = None

    def load_data(self, sig):
        self.sig = sig

    def ret_params(self, Block_Size, Corr_Size, Num_Blocks, Dec_Rate):
        return (int(Block_Size), int(Corr_Size), int(Num_Blocks), int(Dec_Rate))

    def ret_extents(self, xrange, yrange):
        if xrange is None and yrange is None:
            extents = (None, None, None, None)
        elif yrange is None:
            extents = (xrange[0], None, xrange[1], None)
        elif xrange is None:
            extents = (None, yrange[0], None, yrange[1])
        else:
            extents = (xrange[0], yrange[0], xrange[1], yrange[1])

        return extents

    def update(self, Block_Size, Corr_Size, Num_Blocks, Dec_Rate, x_range, y_range):
        (block_size, corr_size, num_blocks, dec_rate) = self.ret_params(Block_Size, Corr_Size, Num_Blocks, Dec_Rate)
        self.update_rngs = False
        if (block_size != self.block_size or corr_size != self.corr_size or num_blocks != self.num_blocks or dec_rate != self.dec_rate):
            self.update_rngs = True
            self.block_size = block_size
            self.corr_size = corr_size
            self.num_blocks = num_blocks
            self.dec_rate = dec_rate

        if self.update_rngs:
            sig = self.sig[::dec_rate]
            timing_est_norm = np.real(sig)
            phase_angle = .1 * np.real(sig)
            timing_est = 2*np.imag(sig)
            auto_corr = .5 * np.imag(sig)

            on_thresh = self.on_thresh * auto_corr
            off_thresh = self.off_thresh * auto_corr
            min_width = int(self.block_size / 2)
            enable_sig = [True if np.abs(value) > .5 else False for value in sig]
            # filter out short enable pulses
            # enable_sig = gen_utils.squelch_short_pulse(enable_sig, min_width)
            # grab the first pulse and find the peak
            max_sig = [True if np.abs(value) > .9 else False for value in sig]

            enable_sig = [1 if value is True else 0 for value in enable_sig]
            max_sig = [1 if value is True else 0 for value in max_sig]
            x_vals = np.arange(0, len(sig), dtype=np.int)
            data_dict = {'sample':x_vals, 'timing_est_norm':timing_est_norm, 'phase_angle':phase_angle,
                         'timing_est':timing_est, 'auto_corr':auto_corr, 'enable_sig':enable_sig, 'max_sig':max_sig,
                         'real':np.real(sig), 'imag':np.imag(sig)}

            try:
                self.df = pd.DataFrame(data_dict)
            except ValueError:
                print(len(x_vals), np.shape(timing_est_norm), np.shape(phase_angle), np.shape(timing_est), np.shape(auto_corr), np.shape(enable_sig), np.shape(max_sig))
                ipdb.set_trace()

            self.time_yrange0 = (1.25*self.df.real.min(), 1.25*self.df.real.max())
            self.time_yrange1 = (1.25*self.df.imag.min(), 1.25*self.df.imag.max())
            # self.time_xrange = (x_vals[0], x_vals[-1])
            self.xrange_new = (x_vals[0], x_vals[-1])
            self.yrange_new = None
            self.est_yrange = (0, 1.2 * np.max(auto_corr))
        else:
            self.xrange_new = x_range
            self.yrange_new = y_range
            self.time_yrange0 = y_range
            self.time_yrange1 = y_range
        print("est updates = {}, {} , {}".format(self.xrange_new, x_range, self.dec_rate))

    def view_est_norm(self):
        opts = dict(width=self.plot_width, height=self.plot_height, xticks=10, yticks=10, xrotation=10, fontsize=dict(xlabel=20, ylabel=20))
        extents = self.ret_extents(self.xrange_new, (0, 1.5))
        plot = hv.Curve(self.df, 'sample', vdims='timing_est_norm', group='Plot', label='Correlation', extents=extents)
        plot = plot.redim.label(timing_est_norm='Normalized Correlation Amplitude')
        plot = plot.redim.label(sample='Sample')
        norm_plot = dynspread(datashade(plot, dynamic=False, min_alpha=self.min_alpha, cmap=Sets1to3[0], x_range=self.xrange_new, y_range=(0, 1.5)).options(**opts))
        enable_plot = hv.Curve(self.df, 'sample', vdims='enable_sig', group='Plot', label='Correlation', extents=extents)
        enable_plot_ds = dynspread(datashade(enable_plot, dynamic=False, min_alpha=self.min_alpha, cmap=Sets1to3[1], x_range=self.xrange_new, y_range=(0, 1.5)).options(**opts))
        max_plot = hv.Curve(self.df, 'sample', vdims='max_sig', group='Plot', label='Correlation', extents=extents)
        max_plot_ds = dynspread(datashade(max_plot, dynamic=False, min_alpha=self.min_alpha, cmap=Sets1to3[2], x_range=self.xrange_new, y_range=(0, 1.5)).options(**opts))
        color_key = [('Timing Est Norm', Sets1to3[0]), ('Packet Detect', Sets1to3[1]), ('Max Value', Sets1to3[2])]
        color_points = hv.NdOverlay({k: hv.Points([0,0], label=str(k)).options(color=v) for k, v in color_key})
        overlay = norm_plot * enable_plot_ds * max_plot_ds * color_points
        overlay = overlay.redim.label(sample='Sample')
        overlay = overlay.redim.label(y='Normalized Correlation Amplitude')
        return norm_plot * enable_plot_ds * max_plot_ds * color_points

    def view_time_est(self):

        # self.update(Block_Size, Corr_Size, Num_Blocks, Dec_Rate, x_range, self.yrange_new)
        extents = self.ret_extents(self.xrange_new, self.est_yrange)
        opts = dict(width=self.plot_width, height=self.plot_height, xticks=10, yticks=10, xrotation=10, fontsize=dict(xlabel=20, ylabel=20))
        plot = hv.Curve(self.df, 'sample', vdims='timing_est', group='Plot', label='Correlation', extents=extents)
        plot = plot.redim.label(timing_est='Correlation Amplitude')
        plot = plot.redim.label(sample='Sample')
        # print("time est = {}".format(self.xrange_new))
        return dynspread(datashade(plot, dynamic=False, min_alpha=self.min_alpha, cmap=Sets1to3[0], x_range=self.xrange_new, y_range=self.est_yrange).options(**opts))

    def view_corr(self):
        extents = self.ret_extents(self.xrange_new, self.est_yrange)
        plot = hv.Curve(self.df, 'sample', vdims='auto_corr', group='Plot', label='Correlation', extents=extents)
        # plot = plot.redim.label(auto_corr='Amplitude')
        opts = dict(width=self.plot_width, height=self.plot_height, xticks=10, yticks=10, xrotation=10, fontsize=dict(xlabel=20, ylabel=20))
        # print("corr = {}".format(self.xrange_new))
        return dynspread(datashade(plot, dynamic=False, min_alpha=self.min_alpha, cmap=Sets1to3[1], x_range=self.xrange_new, y_range=self.est_yrange).options(**opts))

    def view_est(self):
        dmap_est = self.view_time_est()
        dmap_corr = self.view_corr()
        color_key = [('Timing Est', Sets1to3[0]), ('Auto Corr', Sets1to3[1])]
        color_points = hv.NdOverlay({k: hv.Points([0,0], label=str(k)).options(color=v) for k, v in color_key})

        overlay = dmap_est * dmap_corr * color_points
        # overlay = overlay.redim.label(y='Correlation Amplitude')
        return overlay

    def view_est_sigs(self, Block_Size, Corr_Size, Num_Blocks, Dec_Rate, x_range, y_range, update=True):
        if update:
            self.update(Block_Size, Corr_Size, Num_Blocks, Dec_Rate, x_range, y_range)
        dm_est = self.view_est()
        dm_est_norm = self.view_est_norm()

        return dm_est_norm + dm_est

    def view_real(self):
        extents = self.ret_extents(self.xrange_new, self.time_yrange0)
        real_curve = hv.Curve(self.df, 'sample', vdims='real', group='Time', label='Real', extents=extents)
        opts = dict(width=self.plot_width, height=self.plot_height, xticks=10, yticks=10, xrotation=10, fontsize=dict(xlabel=20, ylabel=20))
        return dynspread(datashade(real_curve, cmap=Sets1to3[0], min_alpha=self.min_alpha, x_range=self.xrange_new, y_range=self.time_yrange0, dynamic=False).options(**opts))

    def view_imag(self):
        extents = self.ret_extents(self.xrange_new, self.time_yrange1)
        imag_curve = hv.Curve(self.df, 'sample', vdims='imag', group='Time', label='Imag', extents=extents)
        opts = dict(width=self.plot_width, height=self.plot_height, xticks=10, yticks=10, xrotation=10, fontsize=dict(xlabel=20, ylabel=20))
        return dynspread(datashade(imag_curve, cmap=Sets1to3[1], min_alpha=self.min_alpha, x_range=self.xrange_new, y_range=self.time_yrange1, dynamic=False).options(**opts))

    def view_time_sigs(self, Block_Size, Corr_Size, Num_Blocks, Dec_Rate, x_range, y_range, update=True):
        if update:
            self.update(Block_Size, Corr_Size, Num_Blocks, Dec_Rate, x_range, y_range)
        real_dm = self.view_real()
        real_dm = real_dm.redim.label(real='Amplitude')
        real_dm = real_dm.redim.label(sample='Sample')
        imag_dm = self.view_imag()
        imag_dm = imag_dm.redim.label(imag='Amplitude')
        imag_dm = imag_dm.redim.label(sample='Sample')

        return real_dm + imag_dm

    def view_quad(self, Block_Size, Corr_Size, Num_Blocks, Dec_Rate, x_range, y_range):
        row0 = self.view_est_sigs(Block_Size, Corr_Size, Num_Blocks, Dec_Rate, x_range, y_range)
        row1 = self.view_time_sigs(Block_Size, Corr_Size, Num_Blocks, Dec_Rate, x_range, y_range, update=False)
        layout = (row0 + row1).cols(2)
        # ipdb.set_trace()
        print(self.xrange_new)
        layout = layout.redim.range(Sample=self.xrange_new)
        return layout

def plot_params(sig):

    burst_obj = Burst_Detector()
    burst_obj.load_data(sig)
    plot_height = burst_obj.plot_height
    plot_width = burst_obj.plot_width

    options = Burst_Params(name="")  #"Burst Params"
    options_dec = Dec_Params(name="") # Decimation

    rangexy_norm = hv.streams.RangeXY()
    dmap = hv.DynamicMap(lambda **kw : burst_obj.view_quad(**kw), streams=[options, options_dec, rangexy_norm]).opts(norm=dict(framewise=True))

    row1 = pp.Row(options, options_dec)
    row2 = pp.Row(dmap)
    # row3 = pp.Row(dmap_time)

    layout = pp.Column(row1, row2)
    layout.servable()

sig_len = 1000000
input_sig = np.cos(.00005 * np.arange(sig_len)) + 1j * np.sin(.0001 * np.arange(sig_len))

plot_params(input_sig)

[pjvalla]

Were you able to run the snippet?

[pjvalla]

Any update on this?

[philippjfr]

Sorry for the delay, very swamped at the moment. I'll play with it and see what I can come up with now.

[philippjfr]

Adding framewise=True to every definition of opts = dict(...) in the app worked for me.

[philippjfr]

I get a lot of events bouncing around when zooming and panning though.

[pjvalla]

Thanks a lot. Really appreciate the help.

[philippjfr]

Were you able to address the issue? I didn't have time to investigate further but if there's no way to avoid the issues with events bouncing around we should look into that further.

[pjvalla]

Sorry, for taking so long to get back to you. It does seem to jump around a lot and sometimes the zoom is incorrect. More importantly, I am getting this error.

free(): corrupted unsorted chunks

And then it dies. I did add the framewise=True to all of my opts dictionaries. What exactly does this option do under the hood.

[pjvalla]

So this is the original screen

After increasing the decimation, all 4 plots update appropriately. But if I just select the zoom box (without actually selecting a zoom window), I get an weird zoom on the Curve plots at the bottom.

Then after working with it for a while, I will get the following error.

free(): corrupted unsorted chunks
Aborted (core dumped)

[philippjfr]

Under the hood framewise basically computes the extent of your data and then updates the plot's ranges with those extents on every update. I suspect in this case where multiple plots are linked the updates to a range on one plot are triggering events on the frontend, which bounce back and forth ad infinitum.

The seg faults sound familiar although I've not seen that particular message before but it sounds like it might be related to: https://github.com/pyviz/datashader/issues/570

[pjvalla]

I now see what it is doing. It is pulling the y_range from the top-right plot. I will investigate further.

Okay -- The zooming was on my end. Bug introduced trying to force it to generated the correct extents. So the framewise=True did the trick. I will keep an eye on the crashes.

Thanks for all your help.

[pjvalla]

The crashing only seemed to occur when the "extreme" zooming bug was in the code. The vertical zoom in that case was on the order of 1,000,000X. Hopefully, another data point for: pyviz/datashader#570