search

DigitalSlideArchive / digital_slide_archive

The official deployment of the Digital Slide Archive and HistomicsTK.
https://digitalslidearchive.github.io
Apache License 2.0
108 stars 49 forks source link

Memory spike following import #194

Closed cooperlab closed 2 years ago

cooperlab commented 2 years ago

@manthey We sometimes see memory spike following a mass import of files. @dgutman has observed this as well. It seems to originate in the db container and can be fixed by restarting the docker service. Easy enough, but it would be nice to address this.

manthey commented 2 years ago

If it is in the db container, we might be able to ask mongo to limit its memory use. A long time ago, mongo was very greedy with memory; although it is better than it was, it probably could still be tuned.

manthey commented 2 years ago

Mongo server has a setting that might be relevant:

WiredTiger options:
  --wiredTigerCacheSizeGB arg           Maximum amount of memory to allocate
                                        for cache; Defaults to 1/2 of physical
                                        RAM
cooperlab commented 2 years ago

Thank you @manthey.

@dgutman do you have any inputs on this issue?

dgutman commented 2 years ago

So theoretically that's not a bad thing... particularly related to annotations, where more memory should increase the speed/utility of mongo..

On Thu, Feb 24, 2022 at 2:04 PM Lee Cooper @.***> wrote:

Thank you @manthey https://github.com/manthey.

@dgutman https://github.com/dgutman do you have any inputs on this issue?

— Reply to this email directly, view it on GitHub https://github.com/DigitalSlideArchive/digital_slide_archive/issues/194#issuecomment-1050167563, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFODTVKPIUVUYVLHOIPVZDU4Z6L7ANCNFSM5PH2GWZA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.*** com>

-- David A Gutman, M.D. Ph.D. Associate Professor of Neurology Emory University School of Medicine

cooperlab commented 2 years ago

We should find a sensible limit for the default deployment. We are sometimes spiking above 90% RAM which drive the IT monitoring services crazy.

manthey commented 2 years ago

I think some of this was caused by parsing ndpi files as if they were fully valid tiffs. I think it has been addressed. If not, please reopen the issue with any specifics you can provide.