search

node-red / node-red-dashboard

A dashboard UI for Node-RED
Other
1.32k stars 453 forks source link

Suspected Dashboard Memory Leaks #287

Open really-tall-guy opened 7 years ago

really-tall-guy commented 7 years ago

Hello,

We were noticing that our dashboard would crash occasionally. Initially this was once a week but as our system got more complicated progress to once a day, then almost once an hour at it's worst (with lots of elements and tab switching).

We looked into it and noticed that the Raspberry Pi we were using to display the dashboard (generated by a different pi) appeared to suffer a memory leak that would lead to a crash.

We went through all of the dashboard elements (and we had a few ui_templates that didn't help) and some of the tab switching behaviour to try and narrow the problem down. Additionally we created a completely blank flow and dashboard and watched the dev-tools monitor shown below (manually garbage collection at the very end after the automated one).

image

The two main observations were;

The natural garbage collection doesn't seem to get everything and a manual one tends to do a better job, however, both appear to struggle with tab-switching. The first image below is a screen of heavily loaded dashboard switching tabs forward then back indicated by the spikes with the second a zoom of the mostly standard ui-nodes page.

image

image

Any help would be greatly appreciated. Cheers, tall-guy.

Software Versions

Browsers Checked: Chrome: 60.0.3112.113 Chromium: 56.0.2924.84

Hardware:

bobpotter3999 commented 7 years ago

Same issues as @really-tall-guy. Running on IBM Bluemix so unable to do the same type of memory usage analysis but when switching between tabs display will always freeze. Node-Red 0.17.5 Dashboard 2.5.0

Browser: Version Chrome. 61.0.3163.100 (Official Build) (32-bit

bobpotter3999 commented 7 years ago

@really-tall-guy --> Just tried on an older windows 7 machine using Opera.

Version: 48.0.2685.39 (PGO) - Opera is up to date

Stable Windows 7 32-bit

So far doesn't exhibit the same lockup issues as when using windows 10 and Chrome??

bobpotter3999 commented 7 years ago

And finally ..... Now works on Windows 10 again reliably. Only thing I can think off is that when using the 'manage palette' facility in node-red Bluemix to upgrade the version of dashboard it flagged that you must restart node-red. I did manage to restage it using cloud foundry but that didn't fix things. It just now seems to work again so must be something that Bluemix have done their end. Odd. I'll enjoy it while it lasts!

dceejay commented 7 years ago

Hi - what sort of collection of widgets are you using on these pages ? If you are using charts are you limiting / truncating them, either by time or number of points ? Any more clues you can provide to help us replicate this ? Any ideas how to dig deeper into those charts to find the offending object/function call ?

really-tall-guy commented 7 years ago

Hello,

Software development is not my expertise so I am not sure what might be causing it unfortunately. Any suggestions of what/where to look would be appreciated.

To answer your first question: On the first screenshot we are using no widgets, just a completely blank screen, and still seeing a small increase in memory. Manual garbage collection seems to help but otherwise it creeps slowly. The widgets don't appear to have a huge effect on the memory creep. The creep here is not ideal but manageable with a daily reset/reboot.

The biggest cause of memory issues appears to exist when switching tabs using the UI control. It behaves as though the previous tab does not completely close, but when we return to that tab it behaves like a new instance (best seen in the second screenshot). When we are switching tabs, our system usually crashes within 2-4 hours. This appears to occur regardless of widgets used although more widgets leads to a quicker crash.

In most of our displays we are using mainly UI templates. For charting we tend to use an external charting library (eCharts https://ecomfe.github.io/echarts-doc/public/en/index.html) although not on every page. Other than that we use a couple text/labels, and pie charts. Our main page uses around 55 widgets but no js charts other than your pie charts, mostly text and css in UI templates. Our second page has 3 js charts.

To replicate it I guess the easiest way would be to start off with blank page to see if you are experiencing the same slow creep over the course of a few hours. The second point would be to load up a bunch of widgets on two tabs, and just switch between the tabs and see if you also obtain the same memory spikes that we do. Potentially, it's worth filling the pages with all of the same widget to see if there is a significant difference between them.

If there is any way I can assist further, please let me know.

dceejay commented 7 years ago

and it is definitely the browser end that hangs ? Any chance we can see the template code for those extra charts you use ? hopefully you are loading the library once (using a template with "added to head section" ) and then referencing it from the actual chart widgets ? And removing / deleting old / stale data after a while ?

(In meantime I'll set a repeated loop going to force it here and see what happens)

really-tall-guy commented 7 years ago

Addressing points in order:

An example of the ui template codes are as below. I hope it helps, sorry if it's a bit sloppy.

This is the library loading script from one ui_template.

<script type="text/javascript">
    if(typeof echarts == 'undefined'){
        var oScriptElem = document.createElement("script");
        oScriptElem.type = "text/javascript";
        oScriptElem.src = "http://xxx.xxx.xxx.xxx:1880/echarts.js";
        document.head.insertBefore(oScriptElem, document.head.getElementsByTagName("script")[0])
    }
</script>

This is an example of what calls it in another.

<script>
(function(scope) {
    var timer = setInterval(function(){  
    if(!window.echarts){return;}
    clearInterval(timer);
    console.log("echarts_ring");

    var ringChart = echarts.getInstanceByDom(document.getElementById('ringChart')) || echarts.init(document.getElementById('ringChart'));
    var label_line = {normal: {show: false}, emphasis: {show: false}};
    var bg_ring = {normal: {color: '#252525'}, emphasis: {color: '#252525'}};

    var ringOption = {
      //color: ["#2ec7c9",  "#b6a2de", "#5ab1ef", "#ffb980", "#d87a80", "#8d98b3"],
      color: ["#44ff44" , "#ffff44" ,"#ff4444" ,"#ff44ff" ,"#4444ff"],
      center: ['50%', '50%'],
      //animation: false,
      //legend: {show: true},
      tooltip : {
        trigger: 'item',
      },
      series: [{type: 'pie'}]
    };

    ringChart.setOption(ringOption);    
    ringChart.resize;

    scope.$watch('msg', function(msg){
        ringChart.setOption(msg.payload);
    })    

    }, 100);
})(scope);
</script>

<div id="ringChart" style="width:780px; height:518px;"></div>
kanr commented 7 years ago

Hi I wanted to jump in here, I am encountering this issue running node-red in on a raspberry pi with resin.io. Seeing the same memory leakage with the dashboard and would like to help towards the solution.

If anyone has made progress or has suggestions or questions I will be close by, Thanks!

realsquash commented 6 years ago

I believe I'm having the same problem - eventually the Pi runs out of RAM due to the browser taking it all over about a week. I'm running a Dashboard that isn't very complicated. It can be made available over the internet if anyone wants to check it out. I'm running both node-red and chromium-browser on a Pi 3B.

123aaron commented 6 years ago

I encounter this problem when using the UI control to periodically scroll through tabs. My dashboard has three tabs, each with around 7 charts. One tab plots live the outputs, the other two tabs chart aggregate values over longer time periods. My hope was to have the dashboard on display and automatically scroll through each tab after say 1 minute, but doing this causes chrome to crash after <1 day of running. If I turn the UI control off and leave it sitting on one tab only I don't have any issues, so I believe it is the UI control switching tabs that is causing the problem.

Unfortunately I don't know the solution, but would like to hear if anyone has figured out a workaround.

FYI I'm using a windows desktop and chrome.

zosky commented 6 years ago

ive made a dozen template nodes that get fed from kodi, transmission, my bank + some charts and pis for sensors & machines all around the house. (my flows are rate limiting & charts have reasonable date and point maximums). performance in in one tab was fine but visual-density was too much. moving things to different tabs (and navigating via menu or URL) didn't seem to change performance.

it has come to a grinding halt in the last few days ... after connecting a button to a uiControl node ( go to xyz | go back )... now it takes 5-10 sec for ng-click="send({payload: 'go'})" to show up in a debug bode directly connected to the template. i can delete these uiControls to see if it comes back to life.

if i'm on the wrong path i'd really appreciate someone sparring me a few hours

my setup box:

if more data would help, let me know.

image

ulfmartin commented 6 years ago

I have the same issue. With Chromium and with Chrome. I just had to kill chromium in ssh because there was no more memory left on the raspberry as indicated by top. I also noticed, that the memory usage grows with every tab switch. I tried to dig into the code, but I'm not entirely sure what causes it. I don't have a proper JS IDE here, though, to hunt bugs the proper way

raspi-steve commented 6 years ago

Hello everybody,

I am experiencing the same thing. Is there any solution available yet?

ulfmartin commented 6 years ago

So far you can only work around it: either you use a different, custom UI or you restart your server regularly with a cron job. At the moment I do both: I no longer use this UI. Still I open memory hogging websites in Chromium. And since I'm still running out of memory like this, I restart the machine once per night.

Some people say that a solution would be using a different browser, since Chromium is not really designed to be used in a 24/7 kiosk mode (even though it offers this feature).

dceejay commented 6 years ago

To minimise leakage... don't use too many graphs... if you do use them make sure they are sized appropriately... ie don't try and display thousands of points across a 300 pixel wide widget. They seem to be the worst offender... I can run gauges, buttons and indicators for much much longer before anything goes bang.

mtnbrit commented 4 years ago

Not sure if this is the same issue, but im seeing memory issues with Safari on this simple dashboard. What info can I provide to help diagnose the issue?

nr1

leeoniya commented 4 years ago

@dceejay you might consider uPlot for efficient line charts.

https://github.com/leeoniya/uPlot#performance

dceejay commented 4 years ago

Do you fancy writing a pull request for it ?

leeoniya commented 4 years ago

Do you fancy writing a pull request for it ?

i don't use Node-RED myself, so it's going to be more setup than i have time to commit currently, unfortunately.

you may also consider simply upgrading to Chart.js v3 alpha; they've been chasing uPlot's perf with great success (see the bench above, both v2 and v3 are there). this may be a sufficient level of improvement and will reduce how many chart libs you need to track.

dceejay commented 4 years ago

Thanks, sadly we are stuck behind using angular-chartjs (using Angular 1) - so yes it looks like a complete rewrite is required. So uPlot is also an option, just needs someone with time on their hands to help.