search

agerardin / joplin-plugin-knowledge-graph

Notes as nodes. Explore your Joplin knowledge graph.
MIT License
88 stars 5 forks source link

Optimize system resources usage #5

Open shbach opened 2 years ago

shbach commented 2 years ago

I've noticed that enabling this plugin uses a lot of system resources (the fan on my laptop instantly turns on with this plugin) even if it is toggled to not be showing. Could some optimization be done there? For example if the panel is hidden, prevent refreshing the visualization in the background.

NathanAuckett commented 2 years ago

I can confirm this. With Joplin idle and the graph not displaying, it was averaging about 16 percent cpu usage on my laptop. Disabling the plugin returned Joplin to the average idle usage of 0%

This plugin is really cool! Love your work! Just needs a little improvement in the resource usage area while idle before its usable for my personal work flow

shbach commented 2 years ago

Yeah, this one issue is causing me to leave the plugin disabled most of the time because I don't want my laptop fan to constantly be on (since I usually have Joplin open).

agerardin commented 2 years ago

I have done a bit of investigation. I did not notice anything on my desktop but indeed running on my laptop, I do hear the fan from time to time with up to 100% CPU usage. When idle with the plugin on, the process hover around 40% usage and the fan is mostly quiet again.

Screen Shot 2021-12-31 at 11 07 29 AM

Profiling the process, the plugin starts around the 10s mark and yes you can see the cpu usage going up. Interactions later on (around 20s) do not change much the execution profile. We can see most of the time is spent of the rendering script, mostly to draw nodes.

Screen Shot 2021-12-31 at 11 19 12 AM

The allocation/deallocation cycle on the heap does not appear abnormal. At each frame some 100's of Kbs of allocation to update the canvas and GC starts every so often to free memory.

Screen Shot 2021-12-31 at 11 40 41 AM

I can think of many possible optimizations to reduce the amount of work done when drawing (it will be absolutely needed for scaling to larger graphs and avoid lags), but it don't expect it to solve the current issue.

Note that using the graph library examples show the same behavior, even for the toy examples : https://vasturiano.github.io/force-graph/example/highlight/ with the render process using some 40% of resource as well.

The difference is that at least in Chrome, the browser will pause the execution when the tab looses focus. I don't think there is anything built in in the Joplin plugin api to pause a plugin rendering, but I can probably figure out a way to unmount the graph when the plugin panel is not visible so at least resources are freed when the plugin is not actively used.

NathanAuckett commented 2 years ago

@agerardin The un-mounting when the panel is not visible sounds perfect to me. Its expected that running the plugin and rendering all the notes would use up extra resources. It was not expected that it would continue using those resources when the panel is hidden. If you can develop a way to cease the rendering and thus resources when hidden then I'd say that's exactly what we're after here.

Personally I only want to see the graph sometimes anyways. All other times it would be hidden during my work flow. Any further optimizations for larger graphs would be good too of course but the system resources being used when the graph is not even being displayed is the main issue for me at least.

Thanks again for taking the time to develop this! :)

shbach commented 2 years ago

@agerardin The un-mounting when the panel is not visible sounds perfect to me. Its expected that running the plugin and rendering all the notes would use up extra resources. It was not expected that it would continue using those resources when the panel is hidden. If you can develop a way to cease the rendering and thus resources when hidden then I'd say that's exactly what we're after here.

Personally I only want to see the graph sometimes anyways. All other times it would be hidden during my work flow. Any further optimizations for larger graphs would be good too of course but the system resources being used when the graph is not even being displayed is the main issue for me at least.

Agreed, I have the same workflow/use case.

agerardin commented 2 years ago

I have implemented a fix in v.1.0.5. It should be available in the plugin repo anytime soon. The UI nows stop refreshing in the background so there is very little resource usage when the graph is hidden.

Below is the CPU usage for the following scenario : the graph is hidden for 5 seconds, then displayed for 10sec, and then hidden again.

Screen Shot 2022-01-06 at 11 25 30 AM
shbach commented 2 years ago

Thanks, this is a great improvement and when it works the CPU usage is a lot less.

However, there are two scenarios I've noticed where the CPU usage is still high even with the graph toggled to not be showing:

  1. Upon first opening the Joplin app for the first time, even with the graph toggled to be off on start up, the CPU usage is high. I have to first toggle the graph on and then off again before the CPU usage goes down.
  2. If I switch between notes the CPU usage goes up again even with the graph toggled off. I have to then toggle the graph on and off again for the CPU usage to drop.

These are very common scenarios so these need to be fixed otherwise the CPU usage will still almost always be high unless I resort to constantly toggling the graph on and off.

agerardin commented 2 years ago

Wow thanks for another great feedback. I have pushed v1.0.6. It prevents creating the panel at all at startup if graph settings are set to "don't show graph on startup". Switching between notes was still refreshing the data. It does not anymore. I have profile the code and it looks good. I don't see a difference on cpu usages between the graph plugin behind on and hidden or off.

shbach commented 2 years ago

Switching between notes while graph toggled off no longer increases CPU usage which is great. However, I'm noticing a very large delay (8-10 seconds) before the graph is updated if switching between notes while the graph is toggled on. I don't remember there being such a huge delay before. But that might just be my computer and the large number of notes I have (~1000).

Also a very annoying issue with setting "don't show graph on startup" is that the position of the panel is reset whenever Joplin restarts. This means that every time I start Joplin I have to rearrange and resize all my panels which isn't an insignificant amount of time. This needs to be fixed before that setting is usable for me.

agerardin commented 2 years ago

Another important change in the implementation for 1.06 is that notes are now parsed with a markdown parser rather than using regular expressions. That will allow us to build more interesting functionalities moving forward and this how links to section of a note are currently supported. I have not run benchmarks to compare both approaches, but I would have not anticipated such a drastic change in performance. I have not experienced it but my current test graph does not top 300 nodes. The good news is that soon it won't matter much anymore. The joplin data api had severe limitations for detecting deletion which required to pull a lot of data a lot of times (for example pulling ALL the data at each note selection). Updates (in 2.6 I believe) should lead to a much more efficient approach which I hope to implement soon.

For the second part, I was ignoring until now that you could change the layout in joplin! Cool feature indeed. So do you mean you used to be able to persist the panel position and that it does not work anymore when setting "don't show graph on startup"? Please describe in a separate issue and if you can point to discussion on the topics, that would help find a fix faster.

agerardin commented 2 years ago

Switching between notes while graph toggled off no longer increases CPU usage which is great. However, I'm noticing a very large delay (8-10 seconds) before the graph is updated if switching between notes while the graph is toggled on. I don't remember there being such a huge delay before. But that might just be my computer and the large number of notes I have (~1000).

One thing not addressed by my previous comment is the initial load time. I imagine you also get the 8-10sec delay at startup with the parser. There are several options. One is hybrid approach where I could combine fast parsing with regexp and using the parser when needed, one is iterative where I could expand the graph as nodes get parsed. There also may be a way to parallelized some work but I don't know what joplin/electron allows. Stay tuned. The goal is definitely to support knowledge bases as large as possible.

agerardin commented 2 years ago

So much for not running performance benchmarks first. The markdown parser is 200x slower than regexps. I will revert the code back. It will allow parsing note link with elementId, but will also detect links in code blocks unfortunately.

shbach commented 2 years ago

It will allow parsing note link with elementId, but will also detect links in code blocks unfortunately.

Do you mean note links in code blocks? I don't think that's a common scenario so it might not be that big a deal.

The latest update is much faster now.

agerardin commented 2 years ago

Release v1.1.0 should optimize performances very significantly. Redraw operations are kept to a minimum so the graph can be left open even on constrained hardware. Options in the menu prevent painting photons. This can be useful for large graphs (> several thousand nodes & links) Also COOLDOWN_TIME and COOLDOWN_TICKS can be used to reduce the layout computations.

shbach commented 2 years ago

The performance in the latest version is drastically improved! Even with the graph showing my laptop's fan doesn't turn on.