search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 722 forks source link

Supporting Hotspot Large Page Options #8671

Open AlenBadel opened 4 years ago

AlenBadel commented 4 years ago

Supporting Hotspot Large Page Options

Goals

  1. To allow ease of migration from OpenJDK + Hotspot to OpenJDK + OpenJ9, this activity will focus implementing support for Hotspot large page options on OpenJ9

  2. There are a number of inconsistencies with the existing large page -Xlp options (-Xlp, -Xlp<size>, -Xlp:objectheap, and -Xlp:codecache) this issue will resolve these inconsistencies. These inconsistencies will be resolved by:

    • Identifying and fix current Issues with -Xlp options.
    • Deprecate the existing -Xlp large page options.
    • Implement new paging options that are backwards compatible with -Xlp options allowing the -Xlp options to be mapped to the new options until their removal, while preserving legacy behavior.

Parent Issue: https://github.com/eclipse/openj9/issues/4930

Design

Design Doc: https://ibm.box.com/s/rys4fllji6ji0h1a6tvw79j16w8cum5g

Implementation

Parsing

Currently, all -Xlp options parsing is done within the CodeCache[1], and ObjectHeap[2] somewhat redundantly. This needs to be removed in favor for a centralized solution, where the options are parsed within the VM rather than each component that they affect. Each component should only need to read the parsed results within shared data structure, and configure large pages. Thus the parsing code in [1], and [2] will be removed in favor for a centralized VM implementation.

Centralized Parsing

The solution to this issue will be centralized parsing. The VM will parse all large command line arguments(-XX, & -Xlp options) within jvminit.c[3], and save the net result into shared data structure. The net parsing result consists of the combined affect of all active large page options, neglecting those that are overwritten due to precedence rules. The parsing result will be stored within a new struct inside J9JavaVM[4]. We can name this structure J9LargePageOptionsInfo.

J9LargePageOptionsInfo will consist of the following:

BOOLEAN isEnabledForCodeCache /* Enabled LP on CodeCache */
BOOLEAN isEnabledForObjectHeap /* Enabled LP on ObjectHeap */
UDATA pageSizeForCodeCache /* Specified LP size on CodeCache */
UDATA pageSizeForObjectHeap  /* Specified LP size on ObjectHeap */
J9LargePageCautionLevel lpCautionLevel /* Specify if Warnings, Errors, or Neither have been enabled */
UDATA pageTypeForObjectHeap /* Z/OS Only: Specify if user desires a specific page type on ObjectHeap*/

Where J9LargePageCautionLevel is an enum to specify the following states:

J9CautionUnset /* User has not enabled Warnings, or Errors */
J9CautionWarning /* Enabled Warnings */
J9CautionError /* Enabled Errors */

and pageTypeObjectHeap can be the following:

J9PORT_VMEM_PAGE_FLAG_PAGEABLE_PREFERABLE /* User does not specify to explicitly use a page type. */
J9PORT_VMEM_PAGE_FLAG_PAGEABLE /* User explicitly asks for pageable large pages */
J9PORT_VMEM_PAGE_FLAG_FIXED /* User explicitly asks for fixed large pages */

Populating J9LargePageOptionsInfo

Specifically, this section will focus on the implementation of parsing all -XX, and -Xlp large page options within jvminit.c[3]. This is an area of discussion, since we have two possible implementations.

Firstly, to understand what options are passed in and their position among the user arguments each option variation will need to be searched and consumed[10]. The macro FIND_AND_CONSUME will return the right-most index of an option and if a match is not found the index will return -1. The option will also be consumed if it is found. FIND_AND_CONSUME will be called for each option that is being supported, including all newly proposed -XX options and -Xlp options.

Since each -XX option relates to only a specific subset of J9LargePageOptionsInfo, the implementation can be partitioned to focus on one subset at a time.

An example of this to analyze all options that enable, or disable large pages. Namely these are -XX:[+/-]UseLargePages, -XX:[+/-]UseLargePagesCodeCache, -XX:[+/-]UseLargePagesObjectHeap, -Xlp(-Xlp, -Xlp<size>, -Xlp:[codecache/objectheap].

/* -XX:[-/+]UseLargePages[/CodeCache/ObjectHeap] */
{
    IDATA argUseLargePagesEnable = FIND_AND_CONSUME_ARG(EXACT_MATCH, '-XX:+UseLargePages', NULL);
    IDATA argUseLargePagesDisable = FIND_AND_CONSUME_ARG(EXACT_MATCH, '-XX:-UseLargePages', NULL);
    IDATA argUseLargePagesCodeCacheEnable = FIND_AND_CONSUME_ARG(EXACT_MATCH, '-XX:+UseLargePagesCodeCache', NULL);
    IDATA argUseLargePagesCodeCacheDisable = FIND_AND_CONSUME_ARG(EXACT_MATCH, '-XX:-UseLargePagesCodeCache', NULL);
    IDATA argUseLargePagesObjectHeapEnable = FIND_AND_CONSUME_ARG(EXACT_MATCH, '-XX:+UseLargePagesObjectHeap', NULL);
    IDATA argUseLargePagesObjectHeapDisable = FIND_AND_CONSUME_ARG(EXACT_MATCH, '-XX:-UseLargePagesObjectHeap', NULL);

    /* Simplify to Codecache, and ObjectHeap components */
    argUseLargePagesCodeCacheEnable = OMR_MAX(argUseLargePagesCodeCacheEnable, argUseLargePagesEnable);
    argUseLargePagesCodeCacheDisable = OMR_MAX(argUseLargePagesCodeCacheDisable, argUseLargePagesDisable);
    argUseLargePagesObjectHeapEnable = OMR_MAX(argUseLargePagesObjectHeapEnable, argUseLargePagesEnable);
    argUseLargePagesObjectHeapDisable = OMR_MAX(argUseLargePagesObjectHeapDisable, argUseLargePagesDisable);

    /* Apply -Xlp mapping */
    argUseLargePagesCodeCacheEnable = OMR_MAX(argUseLargePagesCodeCacheEnable, argXlpEnableLargePagesCodeCache);
    argUseLargePagesObjectHeapEnable = OMR_MAX(argUseLargePagesObjectHeapEnable, argXlpEnableLargePagesObjectHeap);

    /* Set Lp Status Flags */
    lpInfo->isEnabledForCodeCache = argUseLargePagesCodeCacheEnable > argUseLargePagesCodeCacheDisable;
    lpInfo->isEnabledForObjectHeap = argUseLargePagesObjectHeapEnable > argUseLargePagesObjectHeapDisable;
}

Parsing -Xlp options A contrary to the preceding -Xlp ruins the party, since it may impact all members at the same time. Hence it's best to parse -Xlp, -Xlp<size>, -Xlp:[codecache/objectheap]:pagesize=<size>,[strict/warn],[pageable/nonpageable] ahead of parsing -XX options.

Xlp options will be parsed together. A set of front-facing (-XX large page compatabile) arguments can be used to translate and integrate the xlp options into how we parse the -XX large page options. The Xlp options can be parsed and simplified to the following variables.

IDATA argXlpEnableLargePagesCodeCache = -1;  // Index of right-most Xlp option that enables large pages on the codecache
IDATA argXlpEnableLargePagesObjectHeap = -1; // Index of right-most Xlp option that enables large pages on the objectheap

IDATA argXlpLargePageSizeInBytesCodeCache = -1; // Index of right-most Xlp option specifies a large page size for the codecache
UDATA xlpLargePageSizeCodeCache = -1;           // Value of that page size
IDATA argXlpLargePageSizeInBytesObjectHeap = -1; // Index of right-most Xlp option specifies a large page size for the objectheap
UDATA xlpLargePageSizeObjectHeap = -1;          // Value of that page size

IDATA argXlpPageWarningsEnable = -1;            // Index of right-most Xlp option specifies a "warn"
IDATA argXlpPageErrorsEnable = -1;            // Index of right-most Xlp option specifies a "strict"

#if defined(J9ZOS390)
IDATA argXlpObjectHeapPageType = -1;            // Index of right-most Xlp option specifies a [non]pageable
IDATA xlpObjectHeapPageType = -1;               // The value of non[pageable]
#endif

Specifics of Xlp, and -Xlp<size> are parsed using straight forward memory mapping. -Xlp:[codecache/objectheap]:pagesize=<size>,[strict/warn],[pageable/nonpageable] will need to be parsed by a sub option parser. Thankfully one already exists within the GC[9]. Since -Xlp parsing can be removed from the GC, the xlpSubOptionsParser can be moved to jvminit.c and used to parse any combination of this option.

Similarly, there will be isolated code to handle and populate other members of this instance. Including -XX:LargePageInBytes[/CodeCache/ObjectHeap]=<size>, -XX:[+/-]LargePageWarnings and -XX:[+/-]LargePageErrors, and -XX:zOSLargePagesObjectHeapType=<type>

isEnabledForCodeCache This member will be enabled only when an option that enables large pages on the codecache is either on the right of an option to disable it, or in absence of such option. As shown in the example above, we know the indices of each option variation this can easily be calculated.

isEnabledForObjectHeap The Above can be applied for the ObjectHeap as well.

pageSizeForCodeCache This will be set in accordance with the size specified on the right most index of any option that impacts the codecache page size. Namely, these are -Xlp<Size>, -Xlp:codecache:pagesize<size>, -XX:LargePageInBytes=<size> and -XX:LargePageInBytesCodeCache=<size>.

pageSizeForObjectHeap The Above can be applied for the ObjectHeap as well. Namely, the options that impact the objectheap pagesize are -Xlp<Size>, -Xlp:codecache:pagesize<size>, -XX:LargePageInBytes<size> and -XX:LargePageInBytesObjectHeap=<size>.

lpCautionLevel Warnings, will be enabled if options to enable it are to the right or in the absence of any option to disable it. The same applies to Errors. Options that enable Warnings include: -XX:+LargePageWarnings, -Xlp:[codecache/objectheap]:pagesize=<size>,warn. Options that disable Warnings include: -XX:-LargePageWarnings

Options that enable Errors include: -XX:+LargePageErrors, -Xlp:[codecache/objectheap]:pagesize=<size>,strict Options that disable Errors include: -XX:-LargePageErrors.

(Z/OS only) pageTypeForObjectHeap As, before in the absence of any option to specify the page type this will be set to J9PORT_VMEM_PAGE_FLAG_PAGEABLE_PREFERABLE, which will attempt for pageable variations but will fall back to fixed large pages. Otherwise, if an option that specifies the pagetype (-XX:zOSLargePagesObjectHeap=[pageable/nonpageable], or -XX:objectheap:pagesize=<size>,[non]pageable) exists the page type specified by the right-most index will be saved. (Pageable - J9PORT_VMEM_PAGE_FLAG_PAGEABLE, NonPageable - J9PORT_VMEM_PAGE_FLAG_FIXED).

(Common) Post-Parsing Implementation Details

In both options, the following checks will be necessary:

CodeCache Implementation

All Parsing will be removed from the codecache.[1] Removing unused Xlp JIT .nls macros. Modify, and move to VM .nls for use within the VM. Instead, large page configuration variables will be read from J9LargePageOptionsInfo.[5] How the codecache verifies, and configures the large pages will not change. [6]

ObjectHeap Implementation

All Parsing will be removed from the objectheap. [2] Removing unused Xlp GC .nls macros. Instead, large page configuration variables will be read from J9LargePAgeOptionsInfo. [7] How the objectheap verifies, and configures the large pages will not change, other than to accompany using data extracted from J9LargePAgeOptionsInfo. [8]

Related

An undocumented -Xlp option that manages the gc meta data page size is to be removed. -Xlp:gcmetadata will not be deprecated, rather it will be fully removed. All other -Xlp options, as were covered in this design will be deprecated.

Mainly this is just includes where the option is parsed, as all it's doing is overwriting the current gc metadata page size. https://github.com/eclipse/openj9/blob/491cde3d0b8383a30689fba244f547d6b9315fce/runtime/gc_modron_startup/mmparse.cpp#L815

Removing the capability of the user specifying the meta data page size, means that the metadata page size will always be set to use the system default page size of the system. https://github.com/eclipse/omr/blob/637f10724dd3de1d4845bd2bc38d9c25456941f3/gc/base/GCExtensionsBase.cpp#L129-L131

This validation would not be necessary. https://github.com/eclipse/omr/blob/637f10724dd3de1d4845bd2bc38d9c25456941f3/gc/base/GCExtensionsBase.cpp#L153-L156

Testing Implementation

CodeCache Testing is implemented here: XlpCodeCacheOptionsTestRunner.java

Objectheap Testing is implemented here: XlpOptionsTest.java

These CodeCache/ObjectHeap Tests use -verbose:sizes output to determine what page size was used to verify each large page size test case. As discussed, -verbose:sizes will need to be changed from displaying -Xlp attributes, to attributes that closely model the new -XX proposed options.

Particularity it's purposed that the verbose output be changed from

  -Xlp:objectheap:pagesize=<Page Size Used>,<Page Type Used on Z/OS>     large page size
                  available large page sizes:
                  < List of pages avaialble on obejct heap >
  -Xlp:codecache:pagesize=<Page Size Used>,<Page Type Used on Z/OS>  large page size for JIT code cache
                  available large page sizes for JIT code cache:
                  < List of pages avaialble on code cache>

To:

-XX:[+/-]UseLargePagesCodeCache
   Page Size Used: <Page Size Used By Code Cache>
   Page Type Used: <Page Type used on Z/OS>
   Available Page Sizes on Code Cache:
   <List of avaialble page sizes on the code cache>
-XX:[+/-]UseLargePagesObjectHeap
   Page Size Used: <Page Size Used by Object Heap>
   Page Type Used: <Page type used by Object Heap on Z/OS>
   Available Page Sizes on Object Heap:
   < List of avaialble page sizes on the object heap>

As each the codecache, and object heap test expects the output to be in the former -Xlp form, this needs to be adapted to conform with the verbose layout changes. The test will also need to be adapted to recongize expected warnings when warnings are enabled.

Summary of Changes

Doc Changes Description PR
Add Docs for new -XX Options TBD
Changes to -Xlp, -Xlp:codecache, and -Xlp:objectheap TBD
Test Changes Description PR
Add Tests for new -XX options TBD
Add j9vmemtests to verify omrvmem_find_valid_page_size & omrvmem_default_large_page_size_ex with OMRPORT_VMEM_PAGE_FLAG_PREFER_PAGEABLE https://github.com/eclipse/omr/pull/4918 https://github.com/eclipse/openj9/pull/8791
Related Changes Description PR
Enable XlpOptions, and XlpCodeCache tests https://github.com/eclipse/openj9/pull/8576
Expand XlpCodeCache tests to include variations https://github.com/eclipse/openj9/pull/8025

References and Footnotes

[1] https://github.com/eclipse/openj9/blob/f7c40c6c6ac97c5bdad13336ab2360f3efb9ff8b/runtime/compiler/control/J9Options.cpp#L1465-L1694 [2] https://github.com/eclipse/openj9/blob/c560cd75ea7ca9c9f2442d2eab7b1afa8a15af1e/runtime/gc_modron_startup/mmparse.cpp#L587-L670 [3] https://github.com/eclipse/openj9/blob/6a33c82664bdc5c0ac8860c85ab8fdcf18bee1ec/runtime/vm/jvminit.c#L1614 [4] https://github.com/eclipse/openj9/blob/491cde3d0b8383a30689fba244f547d6b9315fce/runtime/oti/j9nonbuilder.h#L4780 [5] https://github.com/eclipse/openj9/blob/f7c40c6c6ac97c5bdad13336ab2360f3efb9ff8b/runtime/compiler/control/J9Options.cpp#L1458-L1461 [6] https://github.com/eclipse/openj9/blob/f7c40c6c6ac97c5bdad13336ab2360f3efb9ff8b/runtime/compiler/control/J9Options.cpp#L1696-L1784 [7] https://github.com/eclipse/openj9/blob/c560cd75ea7ca9c9f2442d2eab7b1afa8a15af1e/runtime/gc_modron_startup/mmparse.cpp#L583-L584 [8] https://github.com/eclipse/openj9/blob/c560cd75ea7ca9c9f2442d2eab7b1afa8a15af1e/runtime/gc_modron_startup/mmparse.cpp#L674-L719 [9] https://github.com/eclipse/openj9/blob/8e50c296bba256a5280e4529a6995b3e5e200b9e/runtime/gc_modron_startup/mmparse.cpp#L356-L357 [10] Consumed Arguments: Unique command line arguments must be marked as consumed for control purposes. Duplicate arguments that are later re-defined are automatically consumed. Each option will have at most one unconsumed argument that needs to be consumed when that argument is processed.

dmitripivkine commented 4 years ago

In general I believe there is no need to move existing complexity (sometimes bogus and confusing) to new options world. We should try to keep new options set to be clean and transparent

AlenBadel commented 4 years ago

@dmitripivkine The original request to enable warnings with new options. https://github.com/eclipse/openj9/issues/8671#issuecomment-642314807

Adding of XX:[+/-]LargePageWarnings is a requirement. Are you suggesting we keep this as the only publicly documented warnings option? While we parse the -Xlp strict/warn without directly mapping it to any new option.

DanHeidinga commented 4 years ago

We need both the Warning and the Error forms as they are different sets of behaviour - one is identify an LP problem and one is fail to start if an LP problem occurs.

The proposal is to make those apply equally to codecache and object heap rather than adding variants for both. This gives us fewer new options and leaves room to do something different in the future if required.

dmitripivkine commented 4 years ago

8671 (comment)

I believe XX:[+/-]LargePageWarnings controls output only (print warning or not) and has no impact to JVM logic (continue or terminate). This is easy to understand clean behaviour. If we need to support JVM termination on incompatible settings (strict equivalent) we should use another option to do it (like XX:[+/-]LargePageErrors suggested by you).

AlenBadel commented 4 years ago

We need both the Warning and the Error forms as they are different sets of behaviour - one is identify an LP problem and one is fail to start if an LP problem occurs.

The proposal is to make those apply equally to codecache and object heap rather than adding variants for both. This gives us fewer new options and leaves room to do something different in the future if required.

Agreed. To decouple the complexity:

Precedence Rules: Errors will always take precedence over Warnings if both are enabled.

DanHeidinga commented 4 years ago
  • -XX:[+/-]LargePageWarnings Enable/Disable LP Warnings on CodeCache and ObjectHeap

Yes

  • -XX:[+/-]LargePageErrors Enable/Disable LP Errors on CodeCache and ObjectHeap

Yes

  • -Xlp:[codecache/objectheap]:pagesize=<size>,[strict/warn]. Will be parsed individually, and not mapped to any -XX option.

No. These should be mapped and warnings / errors apply to both CodeCache and ObjectHeap. (And yes that's a change in behaviour but it's a pretty unlikely to occur case)

AlenBadel commented 4 years ago

Great. Updated the design to reflect this. I think we're on all the same page - no pun intended.

AlenBadel commented 4 years ago

I welcome anyone with questions or suggestions to come forth. I'm willing to provide further clarifications if anyone has outstanding questions or concerns before moving forward.

AlenBadel commented 4 years ago

In addition to -XX:[+|-]UseNonpageableLargePagesObjectHeap, I think we should add -XX:[+|-]UsePageableLargePagesObjectHeap.

This option would help anyone that is explicitly trying to use pageable large pages. Using this option with -XX:+LargePage[Warnings/Errors] would help users manage and control page type usage on Z/OS.

AlenBadel commented 4 years ago

In order to fully remove parsing within the GC, we will need to address the undocumented option -Xlp:gcmetadata:pagesize=<size>.

This option will be deprecated along with all other -Xlp options. Would there be any interest in creating equivalent -XX options for this and properly documenting these options?

dmitripivkine commented 4 years ago

In order to fully remove parsing within the GC, we will need to address the undocumented option -Xlp:gcmetadata:pagesize=<size>.

This option will be deprecated along with all other -Xlp options. Would there be any interest in creating equivalent -XX options for this and properly documenting these options?

I believe because this option is not documented we don't need deprecate it, even we don't need to keep it. Instead (if we want to keep this functionality) we should provide an alternative way to setup it using new -XX command. So in case if somebody still use it the new -XX option should be used instead

dmitripivkine commented 4 years ago

In addition to -XX:[+|-]UseNonpageableLargePagesObjectHeap, I think we should add -XX:[+|-]UsePageableLargePagesObjectHeap.

I believe this would be very confusing. There are only two types of pages: pageable and non-pageable. So in your suggestion -XX:+UseNonpageableLargePagesObjectHeap means -XX:-UsePageableLargePagesObjectHeap and vice versa. I believe we need only one of them. I would prefer -XX:[+|-]UsePageableLargePagesObjectHeap

pshipton commented 4 years ago

-XX:[+|-]UsePageableLargePagesObjectHeap works for me.

Not that I'm suggesting it's better, but fyi another option is -XX:zOSLargePagesObjectHeap=<pageable|nonpageable>.

AlenBadel commented 4 years ago

I believe this would be very confusing. There are only two types of pages: pageable and non-pageable. So in your suggestion -XX:+UseNonpageableLargePagesObjectHeap means -XX:-UsePageableLargePagesObjectHeap and vice versa. I believe we need only one of them. I would prefer -XX:[+|-]UsePageableLargePagesObjectHeap

Let's consider the following arguments are used on Z/OS:

Scenario A: -XX:+UseLargePages -XX:LargePageInBytes=1M -XX:+UseNonpageableLargePagesObjectHeap -XX:+LargePageWarnings

Configured Pages: [1M Pageable] A Warning would be shown saying that 1M fixed large pages are not configured, and the ObjectHeap would be using 1M pageable instead.

Scenario B: -XX:+UseLargePages -XX:LargePageInBytes=1M -XX:+LargePageWarnings

Configured Pages: [1M Fixed] A warning would not show, because the default behaviour is to seek 1M pageable preferably but use 1M fixed if pageable is not available. See: https://github.com/eclipse/omr/pull/4762 PREFER_PAGEABLE.

The reasoning here is that there's no way for the user to explicitly specify they're looking to use pageable large pages if that's the only type they're looking to use. The default behaviour without specifying a type is to just use pageable preferably with a fall back to the fixed; which is fine if the user doesn't care what type is used.

dmitripivkine commented 4 years ago

The problem I see we should not have two options for the same thing (just inverse). It creates dependability between options (controversial request needs to be controlled). I believe -XX options should be context free as much as possible (independent) except cases when -XX option plays role of sub option (like -XX:+UseLargePages -XX:LargePageInBytes=1M)

dmitripivkine commented 4 years ago

-XX:[+|-]UsePageableLargePagesObjectHeap works for me.

Not that I'm suggesting it's better, but fyi another option is -XX:zOSLargePagesObjectHeap=<pageable|nonpageable>.

Any of this two is ok for me (just tried to avoid "not non-pageable")

AlenBadel commented 4 years ago

-XX:zOSLargePagesObjectHeap=<pageable|nonpageable>

Agreed. This would definitely solve our problem. As long as everyone stays in agreement I'll make that change.

AlenBadel commented 4 years ago

@DanHeidinga @pshipton What's the best way to proceed with -Xlp:gcmetadata?

We've established that it's an undocumented option and is not used by any OpenJ9 test.

Should the option be either removed totally, deprecated, or re-implemented using -XX convention? It's worth nothing that -Xlp:gcmetadata currently supports every sub-option that -Xlp:objectheap supports. This means that we would need multiple -XX options if we choose to re-implement the option.

See https://github.com/eclipse/openj9/issues/8671#issuecomment-649802595

pshipton commented 4 years ago

If @dmitripivkine wants to keep the option, we'll need -XX: options for it.

-XX:[+|-]UseLargePagesGCMetadata -XX:LargePageSizeInBytesGCMetadata=<size> -XX:zOSLargePagesGCMetadata=<pageable|nonpageable>

I expect -XX:+LargePageWarnings, -XX:+LargePageErrors report for GCMetadata as well.

DanHeidinga commented 4 years ago

My preference would be to remove it during this transition unless @dmitripivkine objects. Don't introduce a new form for it and stop recognizing the -Xlp form of the gcmetadata option

dmitripivkine commented 4 years ago

My preference would be to remove it during this transition unless @dmitripivkine objects. Don't introduce a new form for it and stop recognizing the -Xlp form of the gcmetadata option

I agree. I believe there is no use case when allocation of GC Metadata in LP gains measurable performance difference. @amicic Do you agree?

amicic commented 4 years ago

+1

AlenBadel commented 4 years ago

I've went ahead and updated an implementation overview. https://github.com/eclipse/openj9/issues/8671#issue-571666508

@dmitripivkine @pshipton @gita-omr I appreciate any feedback. Particularly, I've specified two approaches to implement central parsing. (1) Iterating though vm arguments left to right and using pattern matching. (2) Using FIND_AND_CONSUME_ARG

Any opinion on which one we should proceed with would be appreciated.

gita-omr commented 4 years ago

I definitely like (1) more. It just seems more intuitive and easier to maintain. Note that (2) not just uses FIND_AND_CONSUME_ARG but needs to group options in some specific groups, find indexes for all options in the group, then carefully compare those indexes to figure out the precedence. Not sure what happens when an option belongs to more than one group, or new option is added.

dmitripivkine commented 4 years ago

Case (1) has hardcoded preference "+" over "-" option. Also using of macro adding an abstract layer for extra functionality potentially: for example hidden (consumed) option handling, mapped option etc.

There is an example of handling -XX +/- option from current code:

            /* -XX commandline option for +/- TransparentHugepage */
            argIndex = FIND_ARG_IN_VMARGS(EXACT_MATCH, VMOPT_XXNOTRANSPARENT_HUGEPAGE, NULL);
            argIndex2 = FIND_ARG_IN_VMARGS(EXACT_MATCH, VMOPT_XXTRANSPARENT_HUGEPAGE, NULL);
            {
                /* Last instance of +/- TransparentHugepage found on the command line wins
                 *
                 * Default to use OMR setting (Enable for all Linux with THP set to madvise)
                 */
                if (argIndex2 > argIndex) {
                    j9port_control(J9PORT_CTLDATA_VMEM_ADVISE_HUGEPAGE, 1);
                } else if (argIndex > argIndex2) {
                    j9port_control(J9PORT_CTLDATA_VMEM_ADVISE_HUGEPAGE, 0);
                }
            }

An essential in this example:

I don't see why not to use the same pattern for new options?

AlenBadel commented 4 years ago

Not sure what happens when an option belongs to more than one group, or new option is added.

As large pages is concerned I cant fathom an option being added that impacts more than one of these functional groups. It's exactly why we added multiple new options. We've designed this so that we are creating building blocks that can build any possible large page scenario. I don't see the need to map a specific profile, or configuration to a single -XX option. That's the main motivation behind moving away from -Xlp.

gita-omr commented 4 years ago

I think handling +/- can be fixed, as well as using macros vs strcmp. I think the main question is: do we want to just have one pass over all the command line arguments and keep setting flags accordingly or we want to create groups of options, look for those options and then compare their indexes? Maybe the latter is the way it's been done historically but I find it a bit complicated.

dmitripivkine commented 4 years ago

I think handling +/- can be fixed, as well as using macros vs strcmp. I think the main question is: do we want to just have one pass over all the command line arguments and keep setting flags accordingly or we want to create groups of options, look for those options and then compare their indexes? Maybe the latter is the way it's been done historically but I find it a bit complicated.

My vote is to keep handling of new -XX options simple. As far as each new option is independent (but might have relation option/sub-option like -XX:+UseLargePages -XX:LargePageInBytes=1M) the handling should be very simple in a new world. There are two areas of application ("object heap" and "code cache") and three groups of options (general LP request, sub-options to specify size/type of page and caution level). The grammar is really simple - and should be significantly less complicated.

I think we should rename -XX:zOSLargePagesObjectHeap=<type> to -XX:zOSLargePagesObjectHeapType=<type> for clarity. Does it make sense?

There is one more aspect: Currently If default page size is "Large" (like 64K on AIX) the only way to request allocation in 4K ("Small") pages is to use -Xlp4k or -Xlp:objectheap:size=4k. This is confusing (and even inconsistent does not allowed on ZOS). I believe it would it be sufficient to use 4K pages if LP is explicitly not enabled (obviously). But should we allow requests for 4K page like -XX:+UseLargePages -XX:LargePageInBytes=4k?

AlenBadel commented 4 years ago

I think we should rename -XX:zOSLargePagesObjectHeap=<type> to -XX:zOSLargePagesObjectHeapType=<type> for clarity. Does it make sense?

I agree, that makes sense. Will make the change to the design now.

Currently If default page size is "Large" (like 64K on AIX) the only way to request allocation in 4K ("Small") pages is to use -Xlp4k or -Xlp:objectheap:size=4k. This is confusing (and even inconsistent does not allowed on ZOS). I believe it would it be sufficient to use 4K pages if LP is explicitly not enabled (obviously). But should we allow requests for 4K page like -XX:+UseLargePages -XX:LargePageInBytes=4k?

This is true not only for AIX, but for other platforms. Historically, when -Xlp is not specified the codecache will always try to allocate a preferred page size for each platform.

https://github.com/eclipse/openj9/blob/270a41478fa4f73089a001892c021c47ef39a5a6/runtime/compiler/control/J9Options.cpp#L1748-L1758

We can resolve this by adding a new option. Something like -XX:[+/-]UsePreferredPages. Mapping the current behaviour to this option is always enabled by default. When the option is disabled, we could instead just use the smallest OS configured page size which is accessilbe via index 0 of j9vmem_supported_page_sizes.

pshipton commented 4 years ago

I'll put my vote in for option 2. It think it's easier to understand, even though there is some extra work to track the indexes in order to support -Xlp. Down the road once -Xlp parsing is removed, we'll be left with a cleaner solution that is similar to all the other parsing.

gita-omr commented 4 years ago

Ok, since we have 3:1 vote for approach (2) I think we can continue implementing it. Hopefully @DanHeidinga agrees with it as well.

AlenBadel commented 4 years ago

I've went ahead and made the following changes to the design/implementation:

AlenBadel commented 4 years ago

Re-iterating an issue that @dmitripivkine brought up:

Currently If default page size is "Large" (like 64K on AIX) the only way to request allocation in 4K ("Small") pages is to use -Xlp4k or -Xlp:objectheap:size=4k. This is confusing (and even inconsistent does not allowed on ZOS). I believe it would it be sufficient to use 4K pages if LP is explicitly not enabled (obviously). But should we allow requests for 4K page like -XX:+UseLargePages -XX:LargePageInBytes=4k?

Would anyone be interested in this proposed solution https://github.com/eclipse/openj9/issues/8671#issuecomment-662538479? Allowing the user to specify a new option like -XX:-UsePreferredPages rather than specifying -XX:+UseLargePages -XX:LargePageInBytes=4k.

pshipton commented 4 years ago

Just noting it should be XX:-UsePreferredPagesObjectHeap, etc. If -XX:+UseLargePages -XX:LargePageInBytes=4k works, I suggest UsePreferredPages not be included in the initial implementation, but added later if desired.

dmitripivkine commented 4 years ago

In theory to use 4k (small) pages we can specify "do not use LP" like -XX:-UseLargePages and I believe it should be enough

AlenBadel commented 4 years ago

In theory to use 4k (small) pages we can specify "do not use LP" like -XX:-UseLargePages and I believe it should be enough

Don't we need to document a default state for each option? I.e by default JVM will be initialized with-XX:-UseLargePages if it is not explicitly enabled by the user.

AlenBadel commented 4 years ago

Preferred page sizes are actually very similar to sizes used by -XX:+UseLagePages.

On the codecache:

Platform -XX:+UseLargePages PreferredPageSize
Z/OS 1MB Pageable 1MB Pageable
AIX/pLinux 64K/16M 64K
XLinux/Windows Usually 2M 2M

The only difference here is that using -XX:+UseLargePages will allow AIX/pLinux to use 16M pages if configured. Would there be any reason not to remove the hard coded preferred pages in favour for enabling -XX:+UseLargePages by default? I.e allowing AIX/pLinux to use up to 16M pages by default.

If we're able to remove these preferred page sizes, then we could describe -XX:-UseLargePages to use 4k pages as @dmitripivkine has suggested here. https://github.com/eclipse/openj9/issues/8671#issuecomment-664616505

Would there be any performance implications?

AlenBadel commented 4 years ago

@dmitripivkine @DanHeidinga Would there be any objection to enabling -XX:+UseLargePages by default?

dmitripivkine commented 4 years ago

@dmitripivkine @DanHeidinga Would there be any objection to enabling -XX:+UseLargePages by default?

Why do you think it is necessary? We have wide list of platforms (Linux X,P,Z; Windows, AIX, OSX, ZOS all of them should be considered 64 or 32 bits). This is list of page sizes used for object heap by default:

#if defined(AIXPPC)
    requestedPageSize = SIXTY_FOUR_KB; /* Use 64K pages for AIX-32 and AIX-64 */
#elif ((defined(LINUX) || defined(OSX)) && (defined(J9X86) || defined(J9HAMMER)))
    requestedPageSize = TWO_MB; /* Use 2M pages for Linux/OSX x86-64 */
#elif (defined(LINUX) && defined(S390))
    requestedPageSize = ONE_MB; /* Use 1M pages for zLinux-31 and zLinux-64 */
#elif defined(J9ZOS390)
    requestedPageSize = ONE_MB; /* Use 1M PAGEABLE for ZOS-31 and ZOS-64 */
    requestedPageFlags = OMRPORT_VMEM_PAGE_FLAG_PAGEABLE;
#endif /* defined(AIXPPC) */

As you see not all platforms are assigned to use LPs for heap. And for some of them we don't want to change default. For example for Linux PPC LE 64 bit small page is set to 64k and large page to 1G and I believe it is not good idea to use 1G pages as default. Another example default LP for AIX is 16m and it will overwrite current default 64k

AlenBadel commented 4 years ago

Linux PPC LE 64 bit small page is set to 64k and large page to 1G

I'm not sure where you're getting the 1G from. https://github.com/eclipse/omr/blob/796f5d0fe03aa59e10febc044acb98a820f3827b/port/linux/omrvmem.c#L1389-L1426

Strictly speaking on -XX:+UseLargePages, which obtains the default value from the PortLibrary API above. Objectheap will use the second smallest configured page size. That is usually 64K, since smallest is always 4K.
Codecache will use the system default set by the OS via _SC_PAGESIZE. Usually 64K.

I sense we can make this work by increasing the threshold for non-executable large pages (Object-Heap) to look for 2M or less on linux instead of the 64K it is using now. Which would match the default object heap page size hard-coded in your comment above.

DanHeidinga commented 4 years ago

Would there be any objection to enabling -XX:+UseLargePages by default?

I'd need to see the performance analysis on each platform before we made this change. We've seen regressions in the past when changing the LP behaviour. See previous work around using madvise for transparent large pages.

AlenBadel commented 4 years ago

Great. I'll set -XX:-UseLargePages by default, it will be defined as using the code cache and object heap "preferred pages", or hard coded values above. I will open up another issue after the initial implementation on what to do with preferred pages. To determine if we can either create a new option to disable, and ignore these values, or if we can remove them totally.

AlenBadel commented 4 years ago

I have a concern on how we currently output the selected, and available page sizes using -verbose:sizes.

  -Xlp:objectheap:pagesize=64K   large page size
                  available large page sizes:
                  64K
  -Xlp:codecache:pagesize=64K    large page size for JIT code cache
                  available large page sizes for JIT code cache:
                  64K

Printing out -Xlp:[objectheap/codecache] is misleading when the option is not specified at all.

I propose that we adapt the following structure:

-XX:[+/-]UseLargePagesCodeCache   // State of CodeCache Large Pages
-XX:LagePageSizeInBytesCodeCache=<size> // Will only print codecache large page size requested if enabled.
   <Print Large Page Size Actually Used>
   <Specify if Warnings, or Errors are enabled.> // I.e Warning or Errors. Will not print if none are specified. 
   <Print Available Page Sizes on CodeCache>

-XX:[+/-]UseLargePagesObjectHeap  // State of ObjectHeap Large Pages
-XX:LagePageSizeInBytesObjectHeap=<size> // Will only print objectheap large page size requested if enabled.
-XX:LargePageType=[non/pageable] // Will only print objectheap large page type requested if enabled.
   <Print Large Page Size/Type Actually Used>
   <Specify if Warnings, or Errors are enabled.> // I.e Warning or Errors. Will not print if none are specified. 
   <Print Available Page Sizes on ObjectHeap>

The above changes will mean we will need to change how we parse test results within XlpOptionsTest[1], and XlpCodeCacheTest[2].

@dmitripivkine Do you think this should be changed?

[1] https://github.com/eclipse/openj9/blob/2f295ea91cdd97ff37eb5d164fc63014ea718bc7/test/functional/VM_Test/src/j9vm/test/xlp/XlpOptionsTest.java [2] https://github.com/eclipse/openj9/blob/2f295ea91cdd97ff37eb5d164fc63014ea718bc7/test/functional/VM_Test/src/j9vm/test/xlpcodecache/XlpCodeCacheOptionsTest.java

dmitripivkine commented 4 years ago

The goal of -verbose:sizes to show chosen / available range of sizes. Whatever we do (even replace some -Xlp commands to any kind of verbose output) sizes must be clear and easy to see. Would you please provide an example of output you suggesting?

AlenBadel commented 4 years ago

Would you please provide an example of output you suggesting?

-XX:+UseLargePagesCodeCache                    State of Large Pages on CodeCache.
-XX:LagePageSizeInBytesCodeCache=16M   Large Page Size Requested on CodeCache.
   Warnings Enabled
   Page Size Used: 64K
   Available Page Sizes on CodeCache:
   64K
-XX:-UseLargePagesCodeCache                    State of Large Pages on CodeCache.
   Page Size Used: 64K
   Available Page Sizes on CodeCache:
   64K

The only output that somewhat doesn't belong is if Warnings/Errors are enabled. It has nothing to do with size.

dmitripivkine commented 4 years ago

My vote would be report actual data only, do not put what is requested. We need output actually selected as well as any other valid choice. Like ' object heap page size selected "A", available "B","C","D" '. I don't think we need to list all set of options can be used as well as how they are treated. This is not a guide for options.

AlenBadel commented 4 years ago

My vote would be report actual data only, do not put what is requested. We need output actually selected as well as any other valid choice. Like ' object heap page size selected "A", available "B","C","D" '. I don't think we need to list all set of options can be used as well as how they are treated. This is not a guide for options.

I agree. Page data should be anchored under a large page option, or it would look totally misaligned.

I.e

-XX:[+/-]UseLargePagesCodeCache
   Page Size Used: 64K
   Available Page Sizes on CodeCache:
   64K

-XX:[+/-]UseLargePages[CodeCache/ObjectHeap] should be the only options specified. Page info that pertains to the size selected, type selected, and available combinations will be printed indented under that option.

dmitripivkine commented 4 years ago

My vote would be report actual data only, do not put what is requested. We need output actually selected as well as any other valid choice. Like ' object heap page size selected "A", available "B","C","D" '. I don't think we need to list all set of options can be used as well as how they are treated. This is not a guide for options.

I agree. Page data should be anchored under a large page option, or it would look totally misaligned.

I.e

-XX:[+/-]UseLargePagesCodeCache
   Page Size Used: 64K
   Available Page Sizes on CodeCache:
   64K

-XX:[+/-]UseLargePages[CodeCache/ObjectHeap] should be the only options specified. Page info that pertains to the size selected, type selected, and available combinations will be printed indented under that option.

@pshipton Any suggestions how to reformat output of -verbose:sizes ?

AlenBadel commented 4 years ago

I've updated https://github.com/eclipse/openj9/issues/8671#issue-571666508 to reflex the new proposed format of -verbose:sizes output, and the test changes necessary to conform to the new output.

Mainly, the test currently parses the verbose output as written in the -Xlp fashion and we would need accompanying changes to how the output is analyzed on both the code cache and object heap tests to parse this new form.

AlenBadel commented 4 years ago

Just to also note, within the object heap Z/OS specifies that pageable 1M large pages should be used. Would it not be more performant to use OMRPORT_VMEM_PAGE_FLAG_PAGEABLE_PREFERABLE instead which ensures that 1M fixed large pages are used if pageable are not configured, rather than downgrading the page size? @dmitripivkine

https://github.com/eclipse/omr/blob/637f10724dd3de1d4845bd2bc38d9c25456941f3/gc/base/GCExtensionsBase.cpp#L145