memory not correct if multi process sharing memory

[Cas-pian]

The memory is not correct if multi process sharing memory. I think we should use pss of memory_full_info, thanks!

[krinsman]

@Cas-pian Could you clarify what is meant by "multi process sharing memory"?

You mean the notebook server is being run by multiple processes, all of which are sharing memory?

PSS is a Linux-only feature so we don't plan to actively support that at present. However, starting with version 0.3.4 and later, I think you could get NBResuse to report what you're asking by setting in your Jupyter Notebook config file

c.ResourceUseDisplay.process_memory_metrics = [
{"name": "memory_info", "attribute": "rss"}, {"name": "memory_full_info", "attribute": "pss"}
]

but we still need to tell Prometheus to automatically produce an endpoint giving that information. Also even then it would be up to you or an extension developer to implement a GUI using that information.

[jkleint]

I also think accounting for shared memory is a good idea. Is there any reason we could not use the more accurate pss or uss on platforms where it is available, and fall back to rss where it is not?

[dl-sdu]

The memory is not correct if multi process sharing memory. I think we should use pss of memory_full_info, thanks!

Same problem

[EnyMan]

This is happening to me when I run a multiprocessing job, multiprocessing.Pool() to be more precise. I assume it's gonna show the same for threads as well. Wouldnt using psutil.virtual_memory() solve this? from the extension info:

from htop