ShokoAnime / ShokoDesktop

Repository for Shoko Desktop
http://shokoanime.com/shokodesktop/
105 stars 25 forks source link

Memory issues #387

Closed SL-Gundam closed 7 years ago

SL-Gundam commented 8 years ago

I've been verifying tvdb crosslinks and correcting where necessary

Every fifth series or so i would need to restart JMM Desktop since JMM Desktop had risen to over 4 GB in memory and becomes extremely sluggish

This is especially an issue with large series like One Piece. (Just open and leave this series a couple times)

It seems JMM Desktop never cleans up its memory and it only keeps rising. Maybe this is a caching mechanismen but there is a limit to where these are usefull

My system has 16GiB of memory so it's not that my system is running out any time soon.

ElementalCrisis commented 8 years ago

Did a little testing, I can confirm the memory issue. My test DB only has 3 series but by simply clicking each one dozens of time I was able to get the memory use up to around 800MB and it never dropped even after no activity for multiple hours. @maxpiva would you be so kind as to look into this?

hidden4003 commented 8 years ago

I confirm. My steps.

  1. Open collection tab
  2. Select All
  3. Series windows loads, press refresh
  4. With each press of refresh button memory usage only grows, Visual studio shows GC is invoked but memory is not freed.
maxpiva commented 8 years ago

All the refresh subsystem will change, into a changetracker. so no more objects destroyed recreated. Stay tuned.

On Sat, Jun 25, 2016 at 12:00 PM, hidden4003 notifications@github.com wrote:

I confirm. My steps.

  1. Open collection tab
  2. Select All
  3. Series windows loads, press refresh
  4. With each press of refresh button memory usage only grows, Visual studio shows GC is invoked but memory is not freed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/japanesemediamanager/jmmclient/issues/387#issuecomment-228548516, or mute the thread https://github.com/notifications/unsubscribe/ABHDbXl6cLYTAX7pWy6VWL1OwvHbOE4Vks5qPUKNgaJpZM4I-DCN .

hidden4003 commented 8 years ago

Nope, issue still there the amount leaked of memory has decreased so there is that.

maxpiva commented 8 years ago

I don't know how we can change this, without a heavy rewrite of JMM Client.

SL-Gundam commented 7 years ago

This issue has become bigger since renaming JMM to Shoko Previously this issue only applied to the Desktop component. Now it also applies to the Server component I'm running Shoko 3.7.0.3 Information below taken from Process Explorer

A newly started Shoko Server starts out with Working set: ~530 MiB Private bytes: ~713 MiB

A newly started Shoko Desktop starts out with Working set: ~319 MiB Private bytes: ~329 MiB

After opening One Piece using search and awaiting it to finish Shoko Server: Working set: ~486 MiB Private bytes: ~750 MiB Peak Working set: ~3595 MiB Peak Private bytes: ~8229 MiB

Shoko Desktop: Working set: ~5390 MiB Private bytes: ~5563 MiB Peak Working set: ~5399 MiB Peak Private bytes: ~5570 MiB

The amount of memory has significantly increased compared to previous tests. What previously required several visits to the series can now be done with one Revisits to the series don't change the numbers very much for the server component, the Desktop component on the other hand increased by 5 GiB the second visit for all mentioned numbers. All in all the loading of One Piece seems to take significantly longer as well compared to JMM Desktop

Really hope this will become a serious priority now

da3dsoul commented 7 years ago

Yeah, well known for desktop. Server huh. Is search all you've tried, or does it happen with other operations as well? I wonder what isn't being garbage collected. @maxpiva any idea what it might be?

SL-Gundam commented 7 years ago

Only tried search untitled

The red rectangle are the sections where its busy opening one piece (in this case second and third visit). This was during the loading of the series, not the search. The search is quick and fast as it should be

ElementalCrisis commented 7 years ago

The memory issue with Desktop is a bit tricky, as @maxpiva mentioned its an issue at the core of Desktop that requires a rewrite into how he handle loading series. In fact this is one of the main reasons we're working on a new client.

However that memory usage with Server is interesting, I could not get it to go above a couple hundred megabytes and that's with it running for over a week and running an import on a couple hundred files.

SL-Gundam commented 7 years ago

I didn't have it either during importing of files. Just when opening a large series like One Piece. One Piece has quite a lot different tvdb-anidb relationships so maybe that plays a role?

In the simplified view the situation is a tiny bit less extreme...

What i find frustrating is that the situation was bad but at least barely usable. Now it has become so much worse that i'm really tempted to go back to JMM

The weird thing. A series like Gintama is fast and good. The only big difference is the amount tvdb relationships (Somebody really went to town with this in One Piece). There were also less of these when i tested this last time in JMM. untitled1 untitled2 untitled3

I'm going to delete those for One Piece and let you know the results

da3dsoul commented 7 years ago

It's because one piece is out of order on TvDB

SL-Gundam commented 7 years ago

I know thats why they exist, that was never the discussion

I just removed all tvdb links and tested again. Now One Piece is fast as hell without major memory issues Adding 10 links makes it slower but 1 GiG memory usage is still acceptable Adding more tvdb links just makes the situation that much worse.

I advise time is put into optimizing tvdb linking usage especially with a lot of different links like in One Piece

Gintama with only 5 tvdb links and 200 epps does not have any issues but One Piece with 700+epps and 50 tvdb links does have issues with the amount of tvdb links as the root cause

Since this is not usable for me i'm going to stop using Shoko.

I hope somewhere in the future this gets improved and the program becomes usable for me again

I'll leave it up to the Shoko team on what to do with this issue

da3dsoul commented 7 years ago

Are you sure it's links, and not the images? Find a show with 700 episodes but fewer links (bleach maybe). You can add series with no files via the new group button

da3dsoul commented 7 years ago

One of the biggest issues with desktop is that it loads all images at once and doesn't GC them

da3dsoul commented 7 years ago

EC said he can reproduce the problem

SL-Gundam commented 7 years ago

I tested with One Piece, Gintama, Bleach without any files connected to the series

Gintama and Bleach respond well enough. They have only one tvdb link One Piece responds slow but workable Memory usage for Shoko Desktop stays below 2-3 gig after switching between the three a couple times but does steadily increase so switching more should still increase it to critical levels as is currently known issue

The real difference i notice is this This is One Piece, as added based on your suggestion, without files: It has some tvdb links but not all that much. This is what is retrieved from the web cache image

Then after importing the One Piece files you get this untitled1 untitled2 untitled3 So why, after importing the files, does Shoko divide the tvdb links per season and create the memory situation which makes the program unusable for One Piece?

When i delete the extra tvdb links and reduce it to one single link, One Piece responds allmost as well as the other 2 series. image You can also see that as every link is removed (see the red rectangle box) that the CPU time and the amount of data retrieved from the database (located on an external machine so the network spikes are MySQL data being retrieved) becomes less and less. image

So Shoko seems to handle more tvdb links worse every link that is added. Maybe a code optimization would be enough or make sure as little tvdb links are created as possible

Also collasping the tvdb link list seems to increase performance by around 30% (measured on the very big list of One Piece tvdb links)

Hope this helps

ElementalCrisis commented 7 years ago

Great question, we don't use seasons like that so no need for it to be split like that.

da3dsoul commented 7 years ago

Yeah we do. It's for handling and (supposedly) speeding up the retrieving of episode info, but there may be an error there. I can look into it some time not now.

ElementalCrisis commented 7 years ago

Ironically the opposite has happened then because before we never did, we only used the approved links.

merlinx2k commented 7 years ago

i guess there is no fix for this yet? the memory problem is realy getting out of hand for me.

running the desktop app for some hours it gets up to 12gb ram. http://i.imgur.com/RWOMkCP.png

after a restart of the app it starts with nearly 2gb of ram, which is in my opinion why to high. http://i.imgur.com/izje1sk.png

and running it for around 30 min it went up to 6gb again. http://i.imgur.com/KaLiqwJ.png

if there is a way to fix this i would love to hear about it, thanks in adavace :)

SL-Gundam commented 7 years ago

Disable the tvdb links. They cause massive memory overhead and slowdowns

The more tvdb links there are the worse it gets

It got so bad for me i went back to AniDB 'O Matic

ElementalCrisis commented 7 years ago

@da3dsoul may have some new ideas but the last time we talked about this issue, it would require a huge refactor to fix what we think the problem is. There might be ways to reduce it but I don't know if we can eliminate it altogether.

SL-Gundam commented 7 years ago

As i remember it, the refactor concerned that memory was not released/reclaimed properly which caused memory to grow without decreasing.

The reason memory grows to extreme sizes are the tvdb links (both for the desktop and server part of the program).

So 2 separate issues 1 - TVDB links require optimizations so that the memory usage per request is reduced 2 - The memory needs to released/reclaimed when no longer needed

Personally the first issue is more important as, if fixed, it also increases the performance of the program itself by great amounts. If that one is fixed the second issue will be less of concern/priority as each request will still increase the memory usage but by far less amounts

da3dsoul commented 7 years ago

I posted this on discord, but not here. Fixing the memory leak shouldn't be too much trouble (it'll still take awhile and be a pain, but that's memory leaks). What isn't plausible until clientV2 is lowering the​ base memory usage. It should only take about 1.5-2GB for my 900 series collection in v1.

da3dsoul commented 7 years ago

https://www.dropbox.com/s/9zmumpszwlws34i/ShokoDesktop.dmw?raw=1

da3dsoul commented 7 years ago

@maxpiva @Cazzar @Rand-Random @hidden4003 or anyone. It's narrowed down in that. I don't know how to fix it, though.

ElementalCrisis commented 7 years ago

Fixed by @da3dsoul and will be available in 3.7.5.0