search

hpc / Spindle

Scalable dynamic library and python loading in HPC environments
Other
96 stars 23 forks source link

Investigate using mmap to remap global executables to local ones. #16

Closed mcfadden8 closed 5 years ago

mcfadden8 commented 6 years ago

This feature is currently available by running with --debug=yes. It is also leaving the first page of the global file mapped while the remaining pages are mapped to the local file. During testing, determine if the entire local file can be used.

Also, determine whether core dumping will work as expected with the text and data remapped in this way.

gpaulsen commented 6 years ago

Sorry if this is a silly question:

Won't this cause a lot more memory usage for shared libraries, shared across many processes on the same node? What's the advantage?

mcfadden8 commented 6 years ago

This is only for the executable file, not libraries.

The loader creates a mapping (via mmap()) to the text/data segments of the executable file in the globally available file (stored in a file system that is shared across all the nodes). The idea is to remap those mappings to the file in the spindle cache (file system local to the node). This way, when those segments are paged in, they will be read from the local file stored locally instead of the shared one.

The implementation of this already exists and is executed when you run with the --debug=yes command line argument. The intention of this ticket is to test this feature further to determine whether we can use it by default.

One of the benefits of doing this is that it will allow readsym() to /proc/<pid>/exe to continue to work for programs that wish to use it.

mcfadden8 commented 6 years ago

@mplegendre and I discovered that we were able to mmap over the entire text and data segment and /proc/<pid>/exe was preserved. I need to test this again on a TOSS-2 system to see if this is something that may have been fixed with later versions of Linux. Just adding a note here so that I don't forget.