search

eclipse / omr

Eclipse OMR™ Cross platform components for building reliable, high performance language runtimes
http://www.eclipse.org/omr
Other
940 stars 394 forks source link

Inconsistent implementation of VM_FLAGS_SUPERPAGE_SIZE_ANY on OSX #5715

Open dmitripivkine opened 3 years ago

dmitripivkine commented 3 years ago

There is an implementation of OSX specific feature "free choice of super page" in Port library currently. An idea is that VM_FLAGS_SUPERPAGE_SIZE_ANY flag can be provided in fd for mmap as request to let kernel decide page size should be used.

However an implementation of this feature is inconsistent with implementations for different platforms and causing problems.

Common rules supported by all platforms: (1) The description of pages available (provisioned) in OS is stored in two arrays: pageSizes[] and correspondent pageFlags[] (pageFlags[] array originally was added to support different types of pages on ZOS, Unix base platforms usually no need to use it). The convention all platforms supports for pageSizes[] is: the elements of this array are sorted in ascended order and terminated by 0. pageFlags[] elements are used in a context of pageSize[] elements only dependably (2) Callers of omrvmem_reserve_memory_ex() expect if allocation request in Large Pages is failed (for example due lack of them in OS) an alternative request for allocation in small pages should be issued immediately internally. It means top level never re-request same size allocation with different options assuming Port library call did it's best already and in the case failure is received it is final.

An implementation of "free choice of super page" feature consists number of hacks done over https://github.com/eclipse/omr/blob/master/port/osx/omrvmem.c

To represent "free choice of super page" the pageSizes[] element is set to 0 with correspondent pageFlag OMRPORT_VMEM_PAGE_FLAG_SUPERPAGE_ANY. So page description arrays looks like pageSizes[4k, 0, 2m, 0] with correspondent pageFlags[0, OMRPORT_VMEM_PAGE_FLAG_SUPERPAGE_ANY, 0, 0]. This violates rule (1) and caused early termination due pageSizes[1] it is 0 in the code searching for valid page sizes in many places. Even worse, "free choice of super page" element is returned as "default Large Page" for platform in omrvmem_default_large_page_size_ex(). Known callers of this function are treated returned page size 0 as a "Large Page is not available" and not ready to take it in this form. There is even inconsistency with this internally in Port library for OSX in omrvmem_find_valid_page_size().

An implementation in omrvmem_reserve_memory_ex() in the case of OMRPORT_VMEM_PAGE_FLAG_SUPERPAGE_ANY request would never try allocation with small pages if failed. This violates rule (2). So if memory with OMRPORT_VMEM_PAGE_FLAG_SUPERPAGE_ANY flag can not be allocated it failed immediately. This is inconsistent with other platforms.

Currently there is no code in OMR itself or in downstream OpenJ9 project are capable to use OSX specific OMRPORT_VMEM_PAGE_FLAG_SUPERPAGE_ANY feature.

Must do:

My personal preference is to remove VM_FLAGS_SUPERPAGE_SIZE_ANY feature at all due messy implementation. Also because it is supported on OSX only and callers for Multiplatform projects should have OSX specific path to use it (I understand the idea was to engage this mode by using return of omrvmem_default_large_page_size_ex() but it is not implemented cleanly and not a case at the moment). I believe usefulness of this feature is questionable.

An alternative is to keep this feature in (but not as default). There is no mechanism in OpenJ9 to use it (and it is not worse to create one because of limited application on OSX only, using 2m Huge pages is a real choice). However it might be called by providing flag OMRPORT_VMEM_PAGE_FLAG_SUPERPAGE_ANY explicitly.

Again, my vote is just eliminate this feature and clean up the code.

I don't know who is author of this implementation (it is part of initial submission). I wondering there are might be reasons for this option I am missing. Also we would appreciate help with provisioning of Huge Pages on OSX. Is there any documents about it?

@bragaigor @amicic @pshipton @0xdaryl @charliegracie

bragaigor commented 3 years ago

Hi, like Dmitri said, there's very few information when it comes to large(super) pages in OSX. I found this source code where we define the super page flags Dmitri mentioned above: https://opensource.apple.com/source/xnu/xnu-6153.81.5/osfmk/mach/vm_statistics.h.auto.html (it's also depended on XNU kernel version, this doc concerns version xnu-6153.81.5). Arrived to this link through https://www.unix.com/man-page/osx/2/mmap/

pshipton commented 3 years ago

Note that pageFlags is not [0, OMRPORT_VMEM_PAGE_FLAG_SUPERPAGE_ANY, 0, 0]. All the unused entries are set to OMRPORT_VMEM_PAGE_FLAG_NOT_USED, which has value 0x1, so the values are [1, OMRPORT_VMEM_PAGE_FLAG_SUPERPAGE_ANY, 1, 0]. If the rule for traversing the array is to check both pageSize and pageFlags for 0, this still works.