Plot: Provide a selection event listener

[Bengt]

• [x] I have searched to see if a similar issue already exists.

Is your feature request related to a problem? Please describe.

My Gradio application needs to respond to a selection made by the user in a Gradio Plot. While other components like textbox trigger a select event when the selected value changes, Plot objects do not offer such a callback.

Describe the solution you'd like

I would like a .select() event listener in the Plot class that triggers when the user selects something in a Plot object.

Additional context

I have tried building a harness to determine which property changes in the plotly plot in the Gradio Plot object, but I did not succeed. Perhaps someone else can use this as a starting point:

from pathlib import Path
from pprint import pprint
from typing import Any

import gradio
import numpy
from numpy import ndarray
from plotly import graph_objects
from plotly.graph_objs import Figure
from scipy.signal import spectrogram
from scipy.signal.windows import blackman

def get_spectrogram_figure(
    *,
    audio_snippet: ndarray,
    sample_rate: int,
) -> graph_objects.Figure:
    nfft: int = 1024  # Number of points in the fft
    window: Any = blackman(nfft)
    frequencies: ndarray
    bins: ndarray
    power_spectral_density: ndarray
    frequencies, bins, power_spectral_density = spectrogram(
        fs=sample_rate,
        x=audio_snippet,
        window=window,
        nfft=nfft,
    )

    with numpy.errstate(divide='ignore'):
        z: ndarray = 10 * numpy.log10(power_spectral_density)

    z = numpy.where(z == float('-inf'), -200, z)

    data_heatmap: graph_objects.Heatmap = graph_objects.Heatmap(
        x=bins,
        y=frequencies,
        z=z,
        colorscale='Viridis',
    )
    layout: graph_objects.Layout = graph_objects.Layout(
        yaxis=dict(title='Frequency'),
        xaxis=dict(title='Time'),

        clickmode='event+select',
        dragmode='select',
        selectdirection='h',

        selectiondefaults=dict(
            opacity=0.5,
            line_width=5,
        ),

        plot_bgcolor='rgba(0, 0, 0, 0)',
        paper_bgcolor='rgba(0, 0, 0, 0)',
        title_font_color='white',
        font_color='white',
    )

    figure: graph_objects.Figure = graph_objects.Figure(
        data=data_heatmap,
        layout=layout,
    )

    # TODO Make the spectrogram figure interactive so that users can draw bounding boxes on it.

    print(dir(figure.layout))

    # figure.layout.on_change(
    #     lambda _: print(figure.layout.selections),
    #     'selections',
    # )

    # figure_layout: graph_objects.Layout = figure.layout
    # selection = figure_layout.Newselection(
    # )

    figure.layout.selections += (
        dict(
            x0=0,
            x1=1,
            y0=0,
            y1=10000,
        ),
    )

    figure.layout.on_change(
        lambda _: print(f'Selections: {figure.layout.selections}'),
        'selections'
    )

    figure.layout.on_change(
        lambda _: print(f'Annotations: {figure.layout.annotations}'),
        'annotations'
    )

    print('figure.layout.to_plotly_json():')
    pprint(
        figure.layout.to_plotly_json()
    )

    for layout_property in dir(figure.layout):
        if layout_property.startswith('_'):
            continue
        if layout_property == 'on_change':
            continue
        if layout_property == 'figure':
            continue
        if layout_property == 'parent':
            continue
        if layout_property == 'plotly_name':
            continue
        if layout_property == 'pop':
            continue
        if layout_property == 're':
            continue
        if layout_property == 'to_plotly_json':
            continue
        if layout_property == 'update':
            continue

        figure_layout_property = getattr(figure.layout, layout_property)
        print(figure_layout_property)

        figure.layout.on_change(
            lambda _: print(f'{layout_property}: {figure_layout_property}'),
            layout_property,
        )

        for layout_property_property in dir(getattr(figure.layout, layout_property)):
            if layout_property_property.startswith('__'):
                continue
            if layout_property_property.startswith('_'):
                continue
            if layout_property_property == 'on_change':
                continue
            if layout_property_property == 'figure':
                continue
            if layout_property_property == 'parent':
                continue
            if layout_property_property == 'plotly_name':
                continue
            if layout_property_property == 'pop':
                continue
            if layout_property_property == 'to_plotly_json':
                continue
            if layout_property_property == 'update':
                continue
            if layout_property_property == 'on_change':
                continue
            if isinstance(figure_layout_property, tuple):
                continue
            if isinstance(figure_layout_property, str):
                continue
            if isinstance(figure_layout_property, list):
                continue

            figure_layout_property_property = getattr(figure_layout_property, layout_property_property)
            print(figure_layout_property_property)

            figure_layout_property.on_change(
                lambda _: print(f'{layout_property_property}: {figure_layout_property_property}'),
                layout_property_property,
            )

            print()

    figure_layout: graph_objects.Layout = figure.layout
    figure_layout.on_change(
       handle_selections_update,
        'selections',
    )

    return figure

def handle_selections_update(selections):
    print(selections)

def handle_load() -> tuple[Figure, str, str]:
    sample_rate: int = 44100
    audio_snippet: ndarray = numpy.random.rand(sample_rate * 3)

    spectrogram_figure: Figure = get_spectrogram_figure(
        audio_snippet=audio_snippet,
        sample_rate=int(sample_rate),
    )

    start_time_text: str = '0'
    end_time_text: str = '3'

    return \
        spectrogram_figure, \
        start_time_text, \
        end_time_text

def get_gradio_application() -> gradio.Blocks:
    with gradio.Blocks() as gradio_application:
        spectrogram_plot: gradio.Plot = gradio.Plot(
            container=False,
        )

        start_time_text: gradio.Text = gradio.Text(
            visible=True,
        )
        end_time_text: gradio.Text = gradio.Text(
            visible=True,
        )

        gradio_application.load(
            fn=handle_load,
            inputs=[],
            outputs=[
                spectrogram_plot,
                start_time_text,
                end_time_text,
            ],
        )

        spectrogram_plot.change(
            handle_selections_update,
        )

    return gradio_application

def main() -> None:
    gradio_application: gradio.Blocks = get_gradio_application()
    gradio_application.queue(
        concurrency_count=1,
        status_update_rate='auto',
        client_position_to_load_data=None,
        default_enabled=None,
        api_open=False,
        max_size=None,
    )
    gradio_application.launch(
        inline=False,
        inbrowser=False,
        share=False,
        debug=False,
        max_threads=40,
        auth=None,
        auth_message=None,
        prevent_thread_lock=False,
        show_error=False,
        server_name='0.0.0.0',
        server_port=None,
        show_tips=False,
        height=50,
        width="100%",
        encrypt=None,
        ssl_keyfile=None,
        ssl_certfile=None,
        ssl_keyfile_password=None,
        ssl_verify=True,
        quiet=False,
        show_api=False,
        file_directories=None,
        allowed_paths=None,
        blocked_paths=None,
        root_path="",
        _frontend=True,
        app_kwargs=None,
    )

if __name__ == "__main__":
    main()

Note that adding a selection works fine, as the output includes a selections field containing my dummy data:

figure.layout.to_plotly_json():
{'clickmode': 'event+select',
 'dragmode': 'select',
 'font': {'color': 'white'},
 'paper_bgcolor': 'rgba(0, 0, 0, 0)',
 'plot_bgcolor': 'rgba(0, 0, 0, 0)',
 'selectdirection': 'h',
 'selectiondefaults': {'line': {'width': 5}, 'opacity': 0.5},
 'selections': [{'x0': 0, 'x1': 1, 'y0': 0, 'y1': 10000}],
 'template': {'data': {'bar': [{'error_x': {'color': '#2a3f5f'},
                                'error_y': {'color': '#2a3f5f'},
                                'marker': {'line': {'color': '#E5ECF6',
                                                    'width': 0.5},
                                           'pattern': {'fillmode': 'overlay',
                                                       'size': 10,
                                                       'solidity': 0.2}},
                                'type': 'bar'}],
                       'barpolar': [{'marker': {'line': {'color': '#E5ECF6',
                                                         'width': 0.5},
                                                'pattern': {'fillmode': 'overlay',
                                                            'size': 10,
                                                            'solidity': 0.2}},
                                     'type': 'barpolar'}],
                       'carpet': [{'aaxis': {'endlinecolor': '#2a3f5f',
                                             'gridcolor': 'white',
                                             'linecolor': 'white',
                                             'minorgridcolor': 'white',
                                             'startlinecolor': '#2a3f5f'},
                                   'baxis': {'endlinecolor': '#2a3f5f',
                                             'gridcolor': 'white',
                                             'linecolor': 'white',
                                             'minorgridcolor': 'white',
                                             'startlinecolor': '#2a3f5f'},
                                   'type': 'carpet'}],
                       'choropleth': [{'colorbar': {'outlinewidth': 0,
                                                    'ticks': ''},
                                       'type': 'choropleth'}],
                       'contour': [{'colorbar': {'outlinewidth': 0,
                                                 'ticks': ''},
                                    'colorscale': [[0.0, '#0d0887'],
                                                   [0.1111111111111111,
                                                    '#46039f'],
                                                   [0.2222222222222222,
                                                    '#7201a8'],
                                                   [0.3333333333333333,
                                                    '#9c179e'],
                                                   [0.4444444444444444,
                                                    '#bd3786'],
                                                   [0.5555555555555556,
                                                    '#d8576b'],
                                                   [0.6666666666666666,
                                                    '#ed7953'],
                                                   [0.7777777777777778,
                                                    '#fb9f3a'],
                                                   [0.8888888888888888,
                                                    '#fdca26'],
                                                   [1.0, '#f0f921']],
                                    'type': 'contour'}],
                       'contourcarpet': [{'colorbar': {'outlinewidth': 0,
                                                       'ticks': ''},
                                          'type': 'contourcarpet'}],
                       'heatmap': [{'colorbar': {'outlinewidth': 0,
                                                 'ticks': ''},
                                    'colorscale': [[0.0, '#0d0887'],
                                                   [0.1111111111111111,
                                                    '#46039f'],
                                                   [0.2222222222222222,
                                                    '#7201a8'],
                                                   [0.3333333333333333,
                                                    '#9c179e'],
                                                   [0.4444444444444444,
                                                    '#bd3786'],
                                                   [0.5555555555555556,
                                                    '#d8576b'],
                                                   [0.6666666666666666,
                                                    '#ed7953'],
                                                   [0.7777777777777778,
                                                    '#fb9f3a'],
                                                   [0.8888888888888888,
                                                    '#fdca26'],
                                                   [1.0, '#f0f921']],
                                    'type': 'heatmap'}],
                       'heatmapgl': [{'colorbar': {'outlinewidth': 0,
                                                   'ticks': ''},
                                      'colorscale': [[0.0, '#0d0887'],
                                                     [0.1111111111111111,
                                                      '#46039f'],
                                                     [0.2222222222222222,
                                                      '#7201a8'],
                                                     [0.3333333333333333,
                                                      '#9c179e'],
                                                     [0.4444444444444444,
                                                      '#bd3786'],
                                                     [0.5555555555555556,
                                                      '#d8576b'],
                                                     [0.6666666666666666,
                                                      '#ed7953'],
                                                     [0.7777777777777778,
                                                      '#fb9f3a'],
                                                     [0.8888888888888888,
                                                      '#fdca26'],
                                                     [1.0, '#f0f921']],
                                      'type': 'heatmapgl'}],
                       'histogram': [{'marker': {'pattern': {'fillmode': 'overlay',
                                                             'size': 10,
                                                             'solidity': 0.2}},
                                      'type': 'histogram'}],
                       'histogram2d': [{'colorbar': {'outlinewidth': 0,
                                                     'ticks': ''},
                                        'colorscale': [[0.0, '#0d0887'],
                                                       [0.1111111111111111,
                                                        '#46039f'],
                                                       [0.2222222222222222,
                                                        '#7201a8'],
                                                       [0.3333333333333333,
                                                        '#9c179e'],
                                                       [0.4444444444444444,
                                                        '#bd3786'],
                                                       [0.5555555555555556,
                                                        '#d8576b'],
                                                       [0.6666666666666666,
                                                        '#ed7953'],
                                                       [0.7777777777777778,
                                                        '#fb9f3a'],
                                                       [0.8888888888888888,
                                                        '#fdca26'],
                                                       [1.0, '#f0f921']],
                                        'type': 'histogram2d'}],
                       'histogram2dcontour': [{'colorbar': {'outlinewidth': 0,
                                                            'ticks': ''},
                                               'colorscale': [[0.0, '#0d0887'],
                                                              [0.1111111111111111,
                                                               '#46039f'],
                                                              [0.2222222222222222,
                                                               '#7201a8'],
                                                              [0.3333333333333333,
                                                               '#9c179e'],
                                                              [0.4444444444444444,
                                                               '#bd3786'],
                                                              [0.5555555555555556,
                                                               '#d8576b'],
                                                              [0.6666666666666666,
                                                               '#ed7953'],
                                                              [0.7777777777777778,
                                                               '#fb9f3a'],
                                                              [0.8888888888888888,
                                                               '#fdca26'],
                                                              [1.0, '#f0f921']],
                                               'type': 'histogram2dcontour'}],
                       'mesh3d': [{'colorbar': {'outlinewidth': 0, 'ticks': ''},
                                   'type': 'mesh3d'}],
                       'parcoords': [{'line': {'colorbar': {'outlinewidth': 0,
                                                            'ticks': ''}},
                                      'type': 'parcoords'}],
                       'pie': [{'automargin': True, 'type': 'pie'}],
                       'scatter': [{'fillpattern': {'fillmode': 'overlay',
                                                    'size': 10,
                                                    'solidity': 0.2},
                                    'type': 'scatter'}],
                       'scatter3d': [{'line': {'colorbar': {'outlinewidth': 0,
                                                            'ticks': ''}},
                                      'marker': {'colorbar': {'outlinewidth': 0,
                                                              'ticks': ''}},
                                      'type': 'scatter3d'}],
                       'scattercarpet': [{'marker': {'colorbar': {'outlinewidth': 0,
                                                                  'ticks': ''}},
                                          'type': 'scattercarpet'}],
                       'scattergeo': [{'marker': {'colorbar': {'outlinewidth': 0,
                                                               'ticks': ''}},
                                       'type': 'scattergeo'}],
                       'scattergl': [{'marker': {'colorbar': {'outlinewidth': 0,
                                                              'ticks': ''}},
                                      'type': 'scattergl'}],
                       'scattermapbox': [{'marker': {'colorbar': {'outlinewidth': 0,
                                                                  'ticks': ''}},
                                          'type': 'scattermapbox'}],
                       'scatterpolar': [{'marker': {'colorbar': {'outlinewidth': 0,
                                                                 'ticks': ''}},
                                         'type': 'scatterpolar'}],
                       'scatterpolargl': [{'marker': {'colorbar': {'outlinewidth': 0,
                                                                   'ticks': ''}},
                                           'type': 'scatterpolargl'}],
                       'scatterternary': [{'marker': {'colorbar': {'outlinewidth': 0,
                                                                   'ticks': ''}},
                                           'type': 'scatterternary'}],
                       'surface': [{'colorbar': {'outlinewidth': 0,
                                                 'ticks': ''},
                                    'colorscale': [[0.0, '#0d0887'],
                                                   [0.1111111111111111,
                                                    '#46039f'],
                                                   [0.2222222222222222,
                                                    '#7201a8'],
                                                   [0.3333333333333333,
                                                    '#9c179e'],
                                                   [0.4444444444444444,
                                                    '#bd3786'],
                                                   [0.5555555555555556,
                                                    '#d8576b'],
                                                   [0.6666666666666666,
                                                    '#ed7953'],
                                                   [0.7777777777777778,
                                                    '#fb9f3a'],
                                                   [0.8888888888888888,
                                                    '#fdca26'],
                                                   [1.0, '#f0f921']],
                                    'type': 'surface'}],
                       'table': [{'cells': {'fill': {'color': '#EBF0F8'},
                                            'line': {'color': 'white'}},
                                  'header': {'fill': {'color': '#C8D4E3'},
                                             'line': {'color': 'white'}},
                                  'type': 'table'}]},
              'layout': {'annotationdefaults': {'arrowcolor': '#2a3f5f',
                                                'arrowhead': 0,
                                                'arrowwidth': 1},
                         'autotypenumbers': 'strict',
                         'coloraxis': {'colorbar': {'outlinewidth': 0,
                                                    'ticks': ''}},
                         'colorscale': {'diverging': [[0, '#8e0152'],
                                                      [0.1, '#c51b7d'],
                                                      [0.2, '#de77ae'],
                                                      [0.3, '#f1b6da'],
                                                      [0.4, '#fde0ef'],
                                                      [0.5, '#f7f7f7'],
                                                      [0.6, '#e6f5d0'],
                                                      [0.7, '#b8e186'],
                                                      [0.8, '#7fbc41'],
                                                      [0.9, '#4d9221'],
                                                      [1, '#276419']],
                                        'sequential': [[0.0, '#0d0887'],
                                                       [0.1111111111111111,
                                                        '#46039f'],
                                                       [0.2222222222222222,
                                                        '#7201a8'],
                                                       [0.3333333333333333,
                                                        '#9c179e'],
                                                       [0.4444444444444444,
                                                        '#bd3786'],
                                                       [0.5555555555555556,
                                                        '#d8576b'],
                                                       [0.6666666666666666,
                                                        '#ed7953'],
                                                       [0.7777777777777778,
                                                        '#fb9f3a'],
                                                       [0.8888888888888888,
                                                        '#fdca26'],
                                                       [1.0, '#f0f921']],
                                        'sequentialminus': [[0.0, '#0d0887'],
                                                            [0.1111111111111111,
                                                             '#46039f'],
                                                            [0.2222222222222222,
                                                             '#7201a8'],
                                                            [0.3333333333333333,
                                                             '#9c179e'],
                                                            [0.4444444444444444,
                                                             '#bd3786'],
                                                            [0.5555555555555556,
                                                             '#d8576b'],
                                                            [0.6666666666666666,
                                                             '#ed7953'],
                                                            [0.7777777777777778,
                                                             '#fb9f3a'],
                                                            [0.8888888888888888,
                                                             '#fdca26'],
                                                            [1.0, '#f0f921']]},
                         'colorway': ['#636efa',
                                      '#EF553B',
                                      '#00cc96',
                                      '#ab63fa',
                                      '#FFA15A',
                                      '#19d3f3',
                                      '#FF6692',
                                      '#B6E880',
                                      '#FF97FF',
                                      '#FECB52'],
                         'font': {'color': '#2a3f5f'},
                         'geo': {'bgcolor': 'white',
                                 'lakecolor': 'white',
                                 'landcolor': '#E5ECF6',
                                 'showlakes': True,
                                 'showland': True,
                                 'subunitcolor': 'white'},
                         'hoverlabel': {'align': 'left'},
                         'hovermode': 'closest',
                         'mapbox': {'style': 'light'},
                         'paper_bgcolor': 'white',
                         'plot_bgcolor': '#E5ECF6',
                         'polar': {'angularaxis': {'gridcolor': 'white',
                                                   'linecolor': 'white',
                                                   'ticks': ''},
                                   'bgcolor': '#E5ECF6',
                                   'radialaxis': {'gridcolor': 'white',
                                                  'linecolor': 'white',
                                                  'ticks': ''}},
                         'scene': {'xaxis': {'backgroundcolor': '#E5ECF6',
                                             'gridcolor': 'white',
                                             'gridwidth': 2,
                                             'linecolor': 'white',
                                             'showbackground': True,
                                             'ticks': '',
                                             'zerolinecolor': 'white'},
                                   'yaxis': {'backgroundcolor': '#E5ECF6',
                                             'gridcolor': 'white',
                                             'gridwidth': 2,
                                             'linecolor': 'white',
                                             'showbackground': True,
                                             'ticks': '',
                                             'zerolinecolor': 'white'},
                                   'zaxis': {'backgroundcolor': '#E5ECF6',
                                             'gridcolor': 'white',
                                             'gridwidth': 2,
                                             'linecolor': 'white',
                                             'showbackground': True,
                                             'ticks': '',
                                             'zerolinecolor': 'white'}},
                         'shapedefaults': {'line': {'color': '#2a3f5f'}},
                         'ternary': {'aaxis': {'gridcolor': 'white',
                                               'linecolor': 'white',
                                               'ticks': ''},
                                     'baxis': {'gridcolor': 'white',
                                               'linecolor': 'white',
                                               'ticks': ''},
                                     'bgcolor': '#E5ECF6',
                                     'caxis': {'gridcolor': 'white',
                                               'linecolor': 'white',
                                               'ticks': ''}},
                         'title': {'x': 0.05},
                         'xaxis': {'automargin': True,
                                   'gridcolor': 'white',
                                   'linecolor': 'white',
                                   'ticks': '',
                                   'title': {'standoff': 15},
                                   'zerolinecolor': 'white',
                                   'zerolinewidth': 2},
                         'yaxis': {'automargin': True,
                                   'gridcolor': 'white',
                                   'linecolor': 'white',
                                   'ticks': '',
                                   'title': {'standoff': 15},
                                   'zerolinecolor': 'white',
                                   'zerolinewidth': 2}}},
 'title': {'font': {'color': 'white'}},
 'xaxis': {'title': {'text': 'Time'}},
 'yaxis': {'title': {'text': 'Frequency'}}}

However, I cannot figure out how to listen to a change event on the selections.

[Bengt]

Hi, @abidlabs!

I don't think that this feature should require a new component, but rather a new select method on the existing Plot class.

[Bengt]

Note that these kinds of selections can be handled in Dash with for interactive graphing:

https://dash.plotly.com/interactive-graphing

[abidlabs]

I'm not sure if our existing gr.Plot class will be able to handle selections -- let's keep both options open right now and we'll think through it when we come around to this.

[Bengt]

Sure. Thanks for reconsidering.

[abidlabs]

Hey! We've now made it possible for Gradio users to create their own custom components -- meaning that you can write some Python and JavaScript (Svelte), and publish it as a Gradio component. You can use it in your own Gradio apps, or share it so that anyone can use it in their Gradio apps. Here are some examples of custom Gradio components:

• A "Rich Textbox" that allows you to write bold/italics/colored text
• A "Folium Map Viewer" component that allows you to use interactive maps

You can see the source code for those components by clicking the "Files" icon and then clicking "src". The complete source code for the backend and frontend is visible. In particular, its very fast if you want to build off an existing component. We've put together a Guide: https://www.gradio.app/guides/five-minute-guide, and we're happy to help. Hopefully this will help address this issue.