Memory issues

[SL-Gundam]

I've been verifying tvdb crosslinks and correcting where necessary

Every fifth series or so i would need to restart JMM Desktop since JMM Desktop had risen to over 4 GB in memory and becomes extremely sluggish

This is especially an issue with large series like One Piece. (Just open and leave this series a couple times)

It seems JMM Desktop never cleans up its memory and it only keeps rising. Maybe this is a caching mechanismen but there is a limit to where these are usefull

My system has 16GiB of memory so it's not that my system is running out any time soon.

[ElementalCrisis]

Did a little testing, I can confirm the memory issue. My test DB only has 3 series but by simply clicking each one dozens of time I was able to get the memory use up to around 800MB and it never dropped even after no activity for multiple hours. @maxpiva would you be so kind as to look into this?

[hidden4003]

I confirm. My steps.

1. Open collection tab
2. Select All
3. Series windows loads, press refresh
4. With each press of refresh button memory usage only grows, Visual studio shows GC is invoked but memory is not freed.

[maxpiva]

All the refresh subsystem will change, into a changetracker. so no more objects destroyed recreated. Stay tuned.

On Sat, Jun 25, 2016 at 12:00 PM, hidden4003 notifications@github.com wrote:

I confirm. My steps.

1. Open collection tab
2. Select All
3. Series windows loads, press refresh
4. With each press of refresh button memory usage only grows, Visual studio shows GC is invoked but memory is not freed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/japanesemediamanager/jmmclient/issues/387#issuecomment-228548516, or mute the thread https://github.com/notifications/unsubscribe/ABHDbXl6cLYTAX7pWy6VWL1OwvHbOE4Vks5qPUKNgaJpZM4I-DCN .

[hidden4003]

Nope, issue still there the amount leaked of memory has decreased so there is that.

[maxpiva]

I don't know how we can change this, without a heavy rewrite of JMM Client.

[SL-Gundam]

This issue has become bigger since renaming JMM to Shoko Previously this issue only applied to the Desktop component. Now it also applies to the Server component I'm running Shoko 3.7.0.3 Information below taken from Process Explorer

A newly started Shoko Server starts out with Working set: ~530 MiB Private bytes: ~713 MiB

A newly started Shoko Desktop starts out with Working set: ~319 MiB Private bytes: ~329 MiB

After opening One Piece using search and awaiting it to finish Shoko Server: Working set: ~486 MiB Private bytes: ~750 MiB Peak Working set: ~3595 MiB Peak Private bytes: ~8229 MiB

Shoko Desktop: Working set: ~5390 MiB Private bytes: ~5563 MiB Peak Working set: ~5399 MiB Peak Private bytes: ~5570 MiB

The amount of memory has significantly increased compared to previous tests. What previously required several visits to the series can now be done with one Revisits to the series don't change the numbers very much for the server component, the Desktop component on the other hand increased by 5 GiB the second visit for all mentioned numbers. All in all the loading of One Piece seems to take significantly longer as well compared to JMM Desktop

Really hope this will become a serious priority now

[da3dsoul]

Yeah, well known for desktop. Server huh. Is search all you've tried, or does it happen with other operations as well? I wonder what isn't being garbage collected. @maxpiva any idea what it might be?

[SL-Gundam]

Only tried search

The red rectangle are the sections where its busy opening one piece (in this case second and third visit). This was during the loading of the series, not the search. The search is quick and fast as it should be

[ElementalCrisis]

The memory issue with Desktop is a bit tricky, as @maxpiva mentioned its an issue at the core of Desktop that requires a rewrite into how he handle loading series. In fact this is one of the main reasons we're working on a new client.

However that memory usage with Server is interesting, I could not get it to go above a couple hundred megabytes and that's with it running for over a week and running an import on a couple hundred files.

[SL-Gundam]

I didn't have it either during importing of files. Just when opening a large series like One Piece. One Piece has quite a lot different tvdb-anidb relationships so maybe that plays a role?

In the simplified view the situation is a tiny bit less extreme...

What i find frustrating is that the situation was bad but at least barely usable. Now it has become so much worse that i'm really tempted to go back to JMM

The weird thing. A series like Gintama is fast and good. The only big difference is the amount tvdb relationships (Somebody really went to town with this in One Piece). There were also less of these when i tested this last time in JMM.

I'm going to delete those for One Piece and let you know the results

[da3dsoul]

It's because one piece is out of order on TvDB

[SL-Gundam]

I know thats why they exist, that was never the discussion

I just removed all tvdb links and tested again. Now One Piece is fast as hell without major memory issues Adding 10 links makes it slower but 1 GiG memory usage is still acceptable Adding more tvdb links just makes the situation that much worse.

I advise time is put into optimizing tvdb linking usage especially with a lot of different links like in One Piece

Gintama with only 5 tvdb links and 200 epps does not have any issues but One Piece with 700+epps and 50 tvdb links does have issues with the amount of tvdb links as the root cause

Since this is not usable for me i'm going to stop using Shoko.

I hope somewhere in the future this gets improved and the program becomes usable for me again

I'll leave it up to the Shoko team on what to do with this issue

[da3dsoul]

Are you sure it's links, and not the images? Find a show with 700 episodes but fewer links (bleach maybe). You can add series with no files via the new group button

[da3dsoul]

One of the biggest issues with desktop is that it loads all images at once and doesn't GC them

[da3dsoul]

EC said he can reproduce the problem

[SL-Gundam]

I tested with One Piece, Gintama, Bleach without any files connected to the series

Gintama and Bleach respond well enough. They have only one tvdb link One Piece responds slow but workable Memory usage for Shoko Desktop stays below 2-3 gig after switching between the three a couple times but does steadily increase so switching more should still increase it to critical levels as is currently known issue

The real difference i notice is this This is One Piece, as added based on your suggestion, without files: It has some tvdb links but not all that much. This is what is retrieved from the web cache

Then after importing the One Piece files you get this So why, after importing the files, does Shoko divide the tvdb links per season and create the memory situation which makes the program unusable for One Piece?

When i delete the extra tvdb links and reduce it to one single link, One Piece responds allmost as well as the other 2 series. You can also see that as every link is removed (see the red rectangle box) that the CPU time and the amount of data retrieved from the database (located on an external machine so the network spikes are MySQL data being retrieved) becomes less and less.

So Shoko seems to handle more tvdb links worse every link that is added. Maybe a code optimization would be enough or make sure as little tvdb links are created as possible

Also collasping the tvdb link list seems to increase performance by around 30% (measured on the very big list of One Piece tvdb links)

Hope this helps

[ElementalCrisis]

Great question, we don't use seasons like that so no need for it to be split like that.

[da3dsoul]

Yeah we do. It's for handling and (supposedly) speeding up the retrieving of episode info, but there may be an error there. I can look into it some time not now.

[ElementalCrisis]

Ironically the opposite has happened then because before we never did, we only used the approved links.

[merlinx2k]

i guess there is no fix for this yet? the memory problem is realy getting out of hand for me.

running the desktop app for some hours it gets up to 12gb ram. http://i.imgur.com/RWOMkCP.png

after a restart of the app it starts with nearly 2gb of ram, which is in my opinion why to high. http://i.imgur.com/izje1sk.png

and running it for around 30 min it went up to 6gb again. http://i.imgur.com/KaLiqwJ.png

if there is a way to fix this i would love to hear about it, thanks in adavace :)

[SL-Gundam]

Disable the tvdb links. They cause massive memory overhead and slowdowns

The more tvdb links there are the worse it gets

It got so bad for me i went back to AniDB 'O Matic

[ElementalCrisis]

@da3dsoul may have some new ideas but the last time we talked about this issue, it would require a huge refactor to fix what we think the problem is. There might be ways to reduce it but I don't know if we can eliminate it altogether.

[SL-Gundam]

As i remember it, the refactor concerned that memory was not released/reclaimed properly which caused memory to grow without decreasing.

The reason memory grows to extreme sizes are the tvdb links (both for the desktop and server part of the program).

So 2 separate issues 1 - TVDB links require optimizations so that the memory usage per request is reduced 2 - The memory needs to released/reclaimed when no longer needed

Personally the first issue is more important as, if fixed, it also increases the performance of the program itself by great amounts. If that one is fixed the second issue will be less of concern/priority as each request will still increase the memory usage but by far less amounts

[da3dsoul]

I posted this on discord, but not here. Fixing the memory leak shouldn't be too much trouble (it'll still take awhile and be a pain, but that's memory leaks). What isn't plausible until clientV2 is lowering the​ base memory usage. It should only take about 1.5-2GB for my 900 series collection in v1.

[da3dsoul]

https://www.dropbox.com/s/9zmumpszwlws34i/ShokoDesktop.dmw?raw=1

[da3dsoul]

@maxpiva @Cazzar @Rand-Random @hidden4003 or anyone. It's narrowed down in that. I don't know how to fix it, though.

[ElementalCrisis]

Fixed by @da3dsoul and will be available in 3.7.5.0