ui-chart: History does not load after node-red restart

[Tragl1968]

Current Behavior

I have a ui-chart which gets with 2 different topics a value (Temp-SOLL & Temp-IST). The chart is running fine. After a while (a few hours) I did a restart of node-red for testing the new datastore in Dashboard 2.0. After restart the graph was empty and startet with the next new value. No historical data.

The ui-chart is used in a subflow. The subflow is used 2 times. Like described in https://dashboard.flowfuse.com/user/subflows.html#config-nodes

Is there a setting in settings.js? Perhaps I missed this setting while updating from Node-Red 1.x to current 4.0.3

Expected Behavior

Getting historical data in an ui-chart after restart of node-red.

Steps To Reproduce

Restarting of my node-red installation will "clean" the graph

Environment

• Dashboard version: 1.17.1
• Node-RED version: 4.0.3
• Node.js version: 20.17.0
• npm version: 10.8.2
• Platform/OS: PI Bullseye (Upgrade von Buster)
• Browser: Chrome Version 129.0.6668.71 (Offizieller Build) (64-Bit)

Have you provided an initial effort estimate for this issue?

I can not provide an initial effort estimate

[joepavitt]

This is the expected behaviour. We store the contents of the chart in Node-RED's memory, it will be cleared on a restart. If you want to persist the data, you'll need to be storing the data in a database.

[Tragl1968]

Thank you very much, I must have misunderstood the documentation. Is node-red-contrib-persist therefore still a common option under Dashboard 2.0?

[joepavitt]

I've not used that library personally, but looking at the documentation, I don't believe it will as it specifies that it only saves the latest msg object. For a chart, you're likely to have n messages building up your data, so that history would be lost.

[Tragl1968]

Yes, but ui-graph from Dashboard v1.x sends the complete series of the graph and therefore you can save the complete series with “persist”. ui-graph from Dashboard v2.0 only sends the last value.

I therefore kindly ask you to turn this post into a feature request. Alternatively, I would be happy to formulate it and place the feature request in the right place.

[joepavitt]

Thanks @Tragl1968 - we do not take that approach with Dashboard 2.0 as there is no special format required to render data. It makes the charts more low-code friendly, as no function nodes are required to transform data in order to draw them onto the chart in the first place, e.g. data coming straight from an SQL node can go straight into the chart.

[Tragl1968]

Thank you for the hint, I have understood. Do you have any advice for me on the best place to start on this topic?

[joepavitt]

We do still support an array of data being sent to the chart. So, you could accumulate that data yourself into a msg.payload after the chart, then then store that using the plugin you shared?

[colinl]

we do not take that approach with Dashboard 2.0 as there is no special format required to render data.

That is not the issue really. The issue is that the D1 chart provides an easy method of persisting the chart contents and refreshing the chart on restart. With D2 the user has to store data in a database or persistent context and manage dropping data out of the database when it drops off the front of the chart. Not simple at all. For those that need it this will be a big pain.

[Tragl1968]

The issue is that the D1 chart provides an easy method of persisting the chart contents and refreshing the chart on restart. With D2 the user has to store data in a database or persistent context and manage dropping data out of the database when it drops off the front of the chart.

Yes, that's what I mean, thank you colinl

[Tragl1968]

We do still support an array of data being sent to the chart

How must the JSON be formatted to get inside the ui-chart correctly?

Thank you so much in advance.

This is my configuration of ui-graph with "Timescale"

This is my configuration of ui-graph with "Linear"

This is my JSON that I'm injecting to ui-graph (1727683200000 == Monday, 30. September 2024 08:00:00)

{
    "series": [
        "Temperature",
        "Humidity"
    ],
    "labels": [
        1727683200000,
        1727686800000,
        1727690400000
    ],
    "mydata": [
        [
            22,
            23,
            24
        ],
        [
            50,
            55,
            60
        ]
    ]
}

This is the graph "Linear" after injection

This is the graph "Timescale" after injection

And this is the JSON outup of ui-graph for both types (Datapoints for Y are missing)

{
    "_event": {
        "type": "click",
        "clientX": 311,
        "clientY": 601,
        "bbox": {
            "x": 97,
            "y": 589.796875,
            "width": 865.5,
            "height": 32,
            "top": 589.796875,
            "right": 962.5,
            "bottom": 621.796875,
            "left": 97
        }
    },
    "_client": {
        "socketId": "8uZCRNkAfS6OhmaFAAAt",
        "socketIp": "192.168.178.56"
    },
    "topic": "",
    "payload": {
        "series": [
            "Temperature",
            "Humidity"
        ],
        "labels": [
            1727683200000,
            1727686800000,
            1727690400000
        ],
        "mydata": [
            [
                22,
                23,
                24
            ],
            [
                50,
                55,
                60
            ]
        ]
    },
    "_msgid": "35f63afa3ace9c1a",
    "_datapoint": {
        "category": [
            "Temperature",
            "Humidity"
        ],
        "x": [
            1727683200000,
            1727686800000,
            1727690400000
        ],
        "y": [
            null,
            null
        ]
    }

[joepavitt]

Unlike Dashboard 1.0, you can just send the raw-data points, no need to format is specially as per Dashboard 1.0

So in your example, you can have:

[{
    "Temperature": 22,
    "Humidity": 50
    "Timestamp": 1727683200000
}, { ... }]

Then the configuration would be x key as Timestamp, then if you want one line on the chart, you could set y key to Temperature, or if you want multiple, the Series option becomes an array (JSON Type), and you put in ["Temperature", "Humidity"]

[Tragl1968]

Thank you, this way?

My injection

[
    {
        "category": [
            "Temperature",
            "Humidity"
        ],
        "Temperature": 20,
        "Humidity": 54,
        "Timestamp": 1727683200000
    },
    {
        "category": [
            "Temperature",
            "Humidity"
        ],
        "Temperature": 22,
        "Humidity": 52,
        "Timestamp": 1727686800000
    },
    {
        "category": [
            "Temperature",
            "Humidity"
        ],
        "Temperature": 24,
        "Humidity": 50,
        "Timestamp": 1727690400000
    }
]

Leads into

Output of ui-graph

{
    "_event": {
        "type": "click",
        "clientX": 476,
        "clientY": 546,
        "bbox": {
            "x": 413.578125,
            "y": 535.796875,
            "width": 232.328125,
            "height": 16,
            "top": 535.796875,
            "right": 645.90625,
            "bottom": 551.796875,
            "left": 413.578125
        }
    },
    "_client": {
        "socketId": "8uZCRNkAfS6OhmaFAAAt",
        "socketIp": "192.168.178.56"
    },
    "topic": "",
    "payload": [
        {
            "category": [
                "Temperature",
                "Humidity"
            ],
            "Temperature": 20,
            "Humidity": 54,
            "Timestamp": 1727683200000
        },
        {
            "category": [
                "Temperature",
                "Humidity"
            ],
            "Temperature": 22,
            "Humidity": 52,
            "Timestamp": 1727686800000
        },
        {
            "category": [
                "Temperature",
                "Humidity"
            ],
            "Temperature": 24,
            "Humidity": 50,
            "Timestamp": 1727690400000
        }
    ],
    "_msgid": "081ae36834920948",
    "_datapoint": [
        {
            "category": [
                "Temperature",
                "Humidity"
            ],
            "x": 1727683200000,
            "y": [
                20,
                54
            ]
        },
        {
            "category": [
                "Temperature",
                "Humidity"
            ],
            "x": 1727686800000,
            "y": [
                22,
                52
            ]
        },
        {
            "category": [
                "Temperature",
                "Humidity"
            ],
            "x": 1727690400000,
            "y": [
                24,
                50
            ]
        }
    ]
}

[joepavitt]

Sorry, to clarify, the data doesn't need the array in it. On the chart's config, the "Series" option. Change that to JSON type, with the array:

[joepavitt]

Your injection would then just be:

[
    {
        "Temperature": 20,
        "Humidity": 54,
        "Timestamp": 1727683200000
    },
    {
        "Temperature": 22,
        "Humidity": 52,
        "Timestamp": 1727686800000
    },
    {
        "Temperature": 24,
        "Humidity": 50,
        "Timestamp": 1727690400000
    }
]

[Tragl1968]

Thank you so much for your patient, Array works now.

[houser42]

I like many have also been looking for a way to have historical data in ui-chart survive restarts. Did this thread solve that problem for you? If so would you consider sharing your working flow please?

I have a subflow that works well to achieve this with dash V1 here: https://github.com/FlowFuse/node-red-dashboard/issues/1362 but does not work with dash V2.

[Tragl1968]

Did this thread solve that problem for you?

No. This thread #1352 does not solve my problem

If so would you consider sharing your working flow please?

I'm sadly not skilled as needed for solving the request "Archiving historical data of ui-chart"

I would like to hear that #1362 will get a solution for the request "Archiving historical data of ui-chart so that data will survive a restart of node-red service".