Waveform visualization

[ilmoro90]

Hi, I'm trying to visualize a waveform that is saved inside the EPICS Archiver Appliance as follow:

{
        "meta": {
            "name": "CGX-R-MOD-K400-001:KLY:RF_DIGI:RF_FWD_T0_WFM",
            "PREC": "0"
        },
        "data": [
            {
                "secs": 1655977583,
                "val": [
                    0.004194223,
                    0.0044272738,
                    0.0042949147,
                    0.004148382,
                  ...
                ]
             }
         ]
}

The plugin plot this dataset as a point made of 500 values superimposed and it is the correct behaviour because it has just one timestamp. Anyway it is not the correct way to plot the waveform. What I want to do is to start from that particular timestamp that identify the time at which data are stored and produce a new time vector delta-t spaced. In this way it will be possible to plot the dataset as a classical data type. How i can reach this final goal? Do i have to create another function? Any type of suggestion it will be appreciated. Thank you in advance.

[sasaki77]

Let me summarize the current implementation of handling waveform data. Waveform is handled like a table in the plugin. This table format is called wide format data frame in Grafana. Refer here for more information.

Following is an example of waveform data for PV:NAME PV. Grafana shows this waveform data as individual 3 plots on Time series panel. Therefore, Grafana may show 500 plots in your CGX-R-MOD-K400-001:KLY:RF_DIGI:RF_FWD_T0_WFM case.

time	PV:NAME[0]	PV:NAME[1]	PV:NAME[2]
2020-01-02 03:04:00	1	2	3
2020-01-02 03:05:00	4	5	6
2020-01-02 03:06:00	7	8	9

Let's get back to the issue. I couldn't understand a new time vector delta-t spaced you said. Is it like a Inspect Waveforms view of CS-Studio Data Browser? (See attachment) I'd really appreciated if you told me what you want to do more detail.

[ilmoro90]

Thank you for the answer, Yes i wanna to plot the waveform. Actually the PV is stored by Archiver Appliance as follow:

{
  timestamp: 123456789,
  values: [
    val1, val2, val3, val4, ..., val500
  ]
}

• timestamp: single value that represent the sampling time of the waveform
• array values: vector composed by 500 values

The problem arising is that grafana expect 2 arrays with same dimension, one for time, one for values to correctly plot data. In this scenario I just have 1 value for time and 500 values (so scalar against vector) so i cannot plot using grafana.

The workaround that i'm using now is to create a new time vector starting from the sampling time (the dt-spaced vector) and then pass the 2 array to "JSON" (another plugin for grafana) to "correctly" plot the waveform.

This workaround does not make use of your datasource, so my initial question was: how to use your plugin to plot a dataset defined as before? Do I have to define a function inside your plugin to produce a time vector with same dimension respect to array representing the values? Am I missing something?

I will now attach a screenshot to show you what i got until now using "JSON" plugin for grafana link.

[sasaki77]

You made fields for each dataset with pseudo sampling times to show the waveform plot on Time series panel, right? For example, please assume that we have following waveform data and Grafana time range is from 2020-01-01 00:00:00 to 2020-01-01 01:00:00.

[
{
  timestamp: 1577804410000,
  values: [
    val1_1, val1_2, val1_3, val1_4, ..., val1_361
  ]
},
{
  timestamp: 1577804510000,
  values: [
    val2_1, val2_2, val2_3, val2_4, ..., val2_361
  ]
},
]

In this case, you want to get following datafame to show waveform data as a timeseries data. Is this right?

time	T0	T1
2020-01-01 00:00:00	val1_1	val2_1
2020-01-01 00:00:10	val1_2	val2_2
2020-01-01 00:00:20	val1_3	val2_3
...	...	...
2020-01-01 00:59:50	val1_360	val2_360
2020-01-01 01:00:00	val1_361	val2_361

We have to implement some function to return the response like this. I don't think it's difficult to implement this, but I'm concerned about 2 things.

1. Returning the response with pseudo sampling time is correct way or not. Pseudo sampling times are useful for Time series panel to show the data as timeseries, but they don't come from real data.
2. The response has no PV name. However this is not a major issue. We can provide panel titile instead.

Anyway, I'm trying to implement a function to show waveform with array index x axis. Temporary implementation returns a data frame as following.

index	2020-01-01T00:00:10	2020-01-01T00:01:50
1	val1_1	val2_1
2	val1_2	val2_2
3	val1_3	val2_3
...	...	...
360	val1_360	val2_360
361	val1_361	val2_361

This data frame has no time field. Therefore, we can't show this on Time series panel. I tried to use Plotly panel to show the data. Usage of Plotly panel is more complicated than Time series panel, but we can show complicated data.

I'm attaching waveform visualization with the plugin under development and Plotly panel.

https://user-images.githubusercontent.com/6987021/176377532-7ec1863a-d06c-47b8-ac10-67a449df712d.mp4

I hope this information helps you. Any comments are welcome.

[ilmoro90]

Thank you for your time. We are faceing the same problem with 2 different approaches. You are trying to substantially plot waveform values respect the index of the array, instead i'm trying to plot waveform in timeline using the artifact to reproduce the time array displacing 1ms starting from t0 (sampling time).

I think that the question is: what you need? Personally we need for a plot in time, to see some behaviour on particular waveform. I appreciate your work and my question is, why are you using plotly? It can be integrated inside grafana or it was just an example on what your aim is?

Maybe in your plugin can be implemented both ways to arrive to the 2 different visualization, starting from same dataset and letting the user to choose how to plot it on grafana?

[sasaki77]

i'm trying to plot waveform in timeline using the artifact to reproduce the time array displacing 1ms starting from t0 (sampling time).

Is interval between samples always 1ms even if Grafana time range is wide? I don't think we can plot array data as a waveform in that case if time range is wide, for example 1 hour or 1 day. Is this just a trial implementation?

By the way, I might misunderstand t0. I regarded t0 as a starting time of Grafana time range. Actually, is t0 starting time of each timestamp? For example, 1577804410000 and 1577804510000 are t0 for each datapoint for following waveform data?

[
{
  timestamp: 1577804410000,
  values: [
    val1_1, val1_2, val1_3, val1_4, ..., val1_361
  ]
},
{
  timestamp: 1577804510000,
  values: [
    val2_1, val2_2, val2_3, val2_4, ..., val2_361
  ]
},
]

I think that the question is: what you need? Personally we need for a plot in time, to see some behaviour on particular waveform.

Could you tell me a use case of waveform plot in time? I can't imagine actual use case for now.

why are you using plotly? It can be integrated inside grafana or it was just an example on what your aim is?

Plotly panel that I used for demo video is Grafana community plugin. Yes, It was just an example, but it demonstrated that we can plot the waveform using index field instead of time field. I suppose index of array is important. We can roughly understand waveform of array without index, but index helps us to see the array data in detail.

Maybe in your plugin can be implemented both ways to arrive to the 2 different visualization, starting from same dataset and letting the user to choose how to plot it on grafana?

You said the plugin can provide 2 types of data frame for waveform with time field or with index field, right? It may be true, but I'd like to understand your requirements in detail before the implementation.

[ilmoro90]

Is interval between samples always 1ms even if Grafana time range is wide? I don't think we can plot array data as a waveform in that case if time range is wide, for example 1 hour or 1 day. Is this just a trial implementation?

I choose 1 ms because grafana cannot go lower to microsec. If the time range is set to 1 hour or 1 day it appear as a point in plot and its ok for us.

By the way, I might misunderstand t0. I regarded t0 as a starting time of Grafana time range. Actually, is t0 starting time of each timestamp? For example, 1577804410000 and 1577804510000 are t0 for each datapoint for following waveform data?

In your case t0 is 1577804410000 ms . In my implementation i create dt-spaced vector as:

time	values
1577804410000	val1_1
1577804410001	val1_2
...	...
1577804410360	val1_361

Now the data is ready to be plotted in timeseries.

Could you tell me a use case of waveform plot in time? I can't imagine actual use case for now.

We want to plot in time, even with its unaccuracy, because that waveform come from an event and we wanna correlate in grafana that particular event to other dynamics.

You said the plugin can provide 2 types of data frame for waveform with time field or with index field, right? It may be true, but I'd like to understand your requirements in detail before the implementation.

Yes, we can rearrange data in order to keep information adding even a column for index if we wanna plot the waveform in both styles:

• classical
• timeseries

[sasaki77]

Thank you for the explanation. Probably I understand what you want to do. I'm not sure other users are interested in this format, but I'm considering to integrated this into the plugin.

Could you tell me how to handle val2 in my example case. Should the plugin return as following?

time	values
1577804410000	val1_1
1577804410001	val1_2
...	...
1577804410360	val1_361
1577804510000	val2_1
1577804510001	val2_2
...	...
1577804510360	val2_361

If so, I'm concerned about the situation that the array has a lot of elements. For example, array has 2000 elements and it updates the values at 1 second interval. In this case, we need 2 seconds to show 2000 elements with 1ms dt-spaced vector and former sampling data overlap latter sampling data.

[ilmoro90]

Could you tell me how to handle val2 in my example case. Should the plugin return as following?

Yes, correct for values in the matrix. You will obtain a plot in the same line of the waveform then.

If so, I'm concerned about the situation that the array has a lot of elements. For example, array has 2000 elements and it updates the values at 1 second interval. In this case, we need 2 seconds to show 2000 elements with 1ms dt-spaced vector and former sampling data overlap latter sampling data.

Yes maybe it will be slow, but maybe that kind of plot will be read not live but just offline to analyze particular event in time.

Moreover I think that you can get overlapping situation when you sample a lot of sample for each weaveform. In this particular case we sample 362 point and it is not sufficient to enter in overlapping region.

[sasaki77]

Sorry for my late response. I created PR to support dt-spaced array format.

New arrayFormat function in Options category provides 3 types of array format: timeseries, index, and dt-space. Your request might be achieved with dt-space format. Could you build the plugin with PR branch and check if it achieve what you want to do?

If you can't build the plugin, I'll merge the PR and consider to publish new release to check it.

Each format return as following:

timeseries

time	PV:NAME[0]	PV:NAME[1]	...	PV:NAME[360]
1577804410000	val1_1	val1_2	...	val1_361
1577804510000	val2_1	val2_2	...	val2_361

index

index	2020-01-01T00:00:10.000Z	2020-01-01T00:01:50.000Z
0	val1_1	val2_1
1	val1_2	val2_2
...	...	...
360	val1_361	val2_361

dt-space

time	PV:NAME
1577804410000	val1_1
1577804410001	val1_2
...	...
1577804410360	val1_361
1577804510000	val2_1
1577804510001	val2_2
...	...
1577804510360	val2_361

[sasaki77]

@ilmoro90 Could you check if new arrayFormat function achieve what you want to do?

[sasaki77]

@ilmoro90 I'm merging the PR and closing the issue. If there are any problem, please let me know and reopen the issue.