Slowdown of Cerebro over time

[markmandel]

Anyone else finding that cerebro slows down with time - specifically around typing and displaying of the input?

A restart usually fixes it - and it usually happens about once a day, but it can be irritating.

Ubuntu 14.04, v0.2.7

[alexhx5]

I see the exact same problem with v0.2.7 on Win10. It gets super laggy and slow and I have to restart the app every day, as well.

[alexhx5]

@maximbaz and @KELiON have you guys noticed that as well? Where before it was always very fluent and smooth, now it has a 1-5 seconds input/display lag sometimes. I think I saw an additional render process in task manager, I thought that what's causing the problem, but maybe I was just testing some other build I can't remember.

[maximbaz]

I have two render processes now, like I had back then when we were discussing performance issues. I haven't actually observed this slow-down myself, but I'll keep a watch.

Next time you see the laggy app, could you record the RAM and CPU usage of every of its processes, before you restart and then 5 minutes after you restart, to compare?

[alexhx5]

@maximbaz sure, the next time I see the slowdown I will post the resources usage here, but you see I have been always getting random CPU and RAM usage spikes for the render process every like 20 - 40 minutes. But I can see the slowdowns only in v0.2.7. Not sure how to distinguish the "causation" from "correlation" in this case

[maximbaz]

Also, what plugins do you guys use? In the performance thread I showed that gif plugin caches images ~forever~ for 24 hours, and that directly impacts the amount of consumed RAM. Maybe there are more examples like that... 🤔

[alexhx5]

@maximbaz I'm using only clipboard and the convert color plugins. It's probably not the "plugin sorting" function either, since it didn't make it into the v0.2.7 windows .exe build, as you remember from https://github.com/KELiON/cerebro/issues/212

[markmandel]

I have a few installed:

➜  plugins cat package.json
{"name":"cerebro-plugins","dependencies":{"cerebro-linux-system":"0.2.8","cerebro-timezones":"0.0.4","cerebro-gif":"0.2.1","cerebro-shell":"0.1.4","cerebro-emoj":"0.2.4"}}% 

[alexhx5]

@maximbaz and @KELiON I recorded this GIF image to show you the problem. I typed in "time in NY" and started moving the cursor right after, so you could see how much time it takes to process the input:

Side note: for some strange reason, while the LICEcap app was recording the GIF, 2nd render process of cerebro was using 35% of CPU and went up to 500mb RAM.

As you can see, It displays the 1st character after 1-2 seconds and the rest of the string a few more seconds after that.

When you open v0.2.7 cerebro for the first time everything works fine, but after some time it goes into that slowdown mode and don't get back to normal until you restart the app.

When I open cerebro and type in anything, normally it would use just 1% of CPU for a few seconds, but when the slowdowns appear it uses 12% of CPU every time (1 thread of 8 threaded CPU: 12*8 = ~100%) trying really hard to process the input for some reason.

CPU profile:

[maximbaz]

Wow 😮 I have definitely not seen such a big slowdown. And that happens every time? So if you type "time in NY", wait so long until it appears, then clear the input (or close and open window) and type again "time in NY", it will always be that slow again?

[markmandel]

Yep - that's exactly what I see as well. Once it's like that, it doesn't seem to go away either - I restart Cerebro at that point.

[alexhx5]

@maximbaz Yes, after it goes in that slow mode it takes several seconds to show the results every time. The only way to fix it is to restart the app

[maximbaz]

Guys could you please save the CPU profile results next time and share the file here?

The release build has minified JS, so I'm not sure if this file will be useful, but who knows...

[alexhx5]

Sure. I've restarted the app already, but the next time it goes in that slow mode I will record and save the file of the CPU profile itself.

[KELiON]

@markmandel do you also use Cerebro on windows?

[markmandel]

So I'm on Ubuntu 14.04 - so Linux.

[markmandel]

Same thing today. Left running overnight (as per usual), super slow today.

Tried typing, cpu went up to 8%, never higher.

[beeblebrox3]

I'm having this problem too on my laptop with Elementary OS Freya. Only after the update for the latest version. On my work machine, with Mint 17.1 works a little better.

[juba]

Hi,

I also encounter the slowdown problem very regularly. Here are two CPU profiles, one when Cerebro is very sluggish (delay between each keystroke and answer), and one when it is fast after a simple reload. Both profiles have been recorded while just typing emoji (I've got the emoji plugin installed).

I use Cerebro on Linux (latest KDE Neon).

Hope this can be useful !

slow.cpuprofile.txt fast.cpuprofile.txt

[maximbaz]

cool, thanks @juba, I'll try to have a look at these profiles in the evening

[maximbaz]

Hmm from the first glance none of the calls stand out, it just looks as if the same sequence of commands suddenly takes longer to process.

It doesn't repro on my Linux, so I installed 0.2.7 on another computer running Windows, added a few plugins, will see if I can repro it there 😉.

[alexhx5]

Here's the CPU profile and here's the timeline (record allocation timeline), not sure if it's helpful, but it shows the processing graph. Look how long it takes to process each event (see that timeline "heap" file for details):

Also, sometimes when it goes into that slow down mode, it doesn't clear the previous search results for a minute or so and when you open cerebro again it's still showing you the previous results. Guys, any idea on what was added in 0.2.7 that can potentially cause these type of long processing times?

@maximbaz have you been able to reproduce these slowdowns?

[maximbaz]

I haven't been able to reproduce it on Linux (currently running master branch) or Windows (running official 0.2.7 build). The worst I've seen so far was 1-2 seconds delay when cerebro was showing previous search before it cleared itself, but then I immediately try again and it is already instant.

Thanks for the profiles, I'll try to have another look in the evening.

[maximbaz]

Interesting, Windows build is based on a non-minified code... at least it's easy to read the profile 😉

It's really hard to tell though, I can see that the most time spend in what it seems like communication with plugins (there is a call to eachPlugins up in the call stack), and that this function is enumerating both built-in and external plugins to do something (presumably give them your input and see what they will suggest).

Ideally we solve this by understanding what features of 0.2.7 could have caused this, but without knowing the details of that code very well I personally can think of two vectors for the attack (both are pretty much unsatisfying and require someone brave from you guys to build Cerebro from sources):

• Since it's only reproducing in 0.2.7, find the offending commit using git bisect. I can assist on this, in general the way it works is that you tell it that 0.2.6 was all right, but 0.2.7 is broken, and git will suggest you to build Cerebro from specific commits, and you test to see if you can reproduce the issue or not. Git tells me that there would be around 5 attempts required to identify the commit, assuming Cerebro is getting slow after a day of usage, this gives us 5 days of testing.

• Since the slowdown happens somewhere after enumerating plugins, determine if this is caused by one (or few) specific plugins. Disable all custom ones, see if it's still reproducing (it probably is, as we've already determined that two people in this this thread have completely different set of plugins and both experience the slowdown). Disable some of the core plugins (in app/main/plugins/core/index.js), see if the slowdown is related to any particular plugins. This approach will give us less specific answers (if any) comparing to the previous one.

[SquirrelsMcGee]

I'm also getting this issue on Windows 10 64-bit Home edition

Short video detailing the issue, (Note that, while I was moving the cursor around like a fool, I was typing, and when I stopped moving the cursor, I stopped typing). https://a.safe.moe/j9JAV.webm Video includes live CPU and Memory usage

[nazar-pc]

I'm experiencing this on Ubuntu 17.10 with 0.2.8, when typing first letter one of Cerebro's processes takes 96% of CPU for 1.5-2 seconds, subsequent letters are faster, but still much slower that after fresh start. The only plugins I have installed are:

• Linux System
• Emoj
• Shell

First one is certainly Linux-specific, so it shouldn't be an issue. Will disable Emoj now and check if helps, since some other have this installed as well.

[meguroyama]

I can confirm that this is still happening on w10_x64 build 1493.1066

[nazar-pc]

Still happens without Emoj

[maximbaz]

I'm looking forward to see https://github.com/KELiON/cerebro/issues/277 done, perhaps it's one of the core plugins that causes your issues, and with #277 you'll be able to test this theory.

[nazar-pc]

It looks like there is some memory leak. Even without typing anything by opening and closing Cerebro one of the processes constantly consumes more memory. I see that garbage collector tries to reduce memory footprint, but it is still growing slowly.

Still trying to identify what exactly causes this issue: often usage or just general number of pressed keys and shortcuts in OS. It doesn't seem to be related to how long application is running.

[Stanzilla]

I noticed this, too. When I then end Cerebro it crashes with the following message:

[Haarolean]

Happens on linux with v0.2.7 still.

[nazar-pc]

With 0.3.0-alpha I've disabled a few plugins (I believe Math and Google search) and let it run. It still becomes slower over time, but nowhere near the extend it did before, it remains pretty responsive even after being running for more than a day without interruption.

So there is a positive progress here, which is nice to see 👍

[KELiON]

@nazar-pc let's continue to look on it, but if it works better now – probably it was an electron issue, not plugin's responsibility. Because neither google nor math doesn't run any tasks in background.

[SquirrelsMcGee]

Possible relevance?

[Stanzilla]

For me it gets REALLY slow after a while only when typing, bringing up the UI is still instant but if you start typing it sometimes takes 5s+ for the letters to appear in the box

[Stanzilla]

CPU-20170802T184842.zip

TimelineRawData-20170802T185332.zip

[Stanzilla]

@KELiON do you have time to investigate this? It's super annoying and I have to restart Cerebro daily because of it.

[gepd]

Same here with windows 10 64b.

I've uninstalled all plugins and let only the Basic App and the problem is the same

[Haarolean]

0.3.0 on windows is lagging from the start, takes more than 200mb of RAM.

[nazar-pc]

I've prepared a set of patches that were merged in about 13 days ago. You can see them linked higher. Those patches should change search performance dramatically in a good way. Unfortunately, @KELiON didn't respond whether he'll release a new Cerebro version soon, so I encourage you to try building new Cerebro version with those patches yourself and report if it works better for you.

[maximbaz]

He did release cerebro-basic-apps v1.0.11 after merging all your patches 🙂 You don't have to recompile Cerebro itself to start using the new version of cerebro-basic-apps, the plugin is upgraded automatically 🙂

[nazar-pc]

Indeed, thanks @maximbaz! @Haarolean, you're not using latest version, upgrade to 0.3.1 and upgrade all plugins.

[Haarolean]

@nazar-pc it shows different versions in different places. Right clicking on tray icon shows it's 0.3.1, but 0.3.0 in app itself lol. Also, considering I'm on 0.3.1, it's still utter shit, lagging and other stuff The blank 'window' is cerebro itself.

[binarymelon]

@Haarolean I'm not sure the blank window is specifically the fault of Cerebro. I've been having the same behaviour with multiple apps (cerebro, teams, discord). I haven't found the culprit yet, but I suspect something to do with geforce experience (do you have this installed?).

[Haarolean]

@ryanproud I do but that doesn't happen to my discord.

[nazar-pc]

@Haarolean, you have latest version of Cerebro itself, I've seen the same bug with version number myself. Do you have the latest version of cerebro-basic-apps too?

I'm not sure what is the reason of this bug, but the fix I've made to search mechanism should improve its performance by an order of a magnitude. My uptime is currently under 1.5 days and it still works instantly. Will debug more when/if it will become slow again (previously took several days, my typical uptime is 7-8 days).

[lorenz]

I'm running it on the open-source Nouveau driver on Linux and I'm experienceing the same, not happening to VS Code and similar Chromium-based apps. So I doubt that anything specific to Nvidia.

[maximbaz]

Have you tried starting Cerebro with a flag cerebro --disable-gpu ? I use it always with this flag, saves a couple of hundred megabytes of RAM, haven't seen any issue with it :wink:

[lorenz]

Hmm, seems to perform better (at least initially). Still weird that it would do that with pretty much all GPU drivers.

[nazar-pc]

I've been testing this for a while, last uptime:

nazar-pc@nazar-pc ~> uptime
 04:32:21 up 6 days,  3:39,  1 user,  load average: 0,95, 0,97, 0,86

It still works instantly! I believe this particular issue can be closed safely and other issues should be reported separately.

@markmandel, it is working fine for you too?

UPD: Still instant:

nazar-pc@nazar-pc ~> uptime
 04:50:41 up 10 days,  3:58,  1 user,  load average: 2,68, 2,45, 1,98