Better batch system directives configuration.

[hjoliver]

The current suite.rc spec doesn't make it easy to switch between different batch systems, because there is only one directives section, which is unlikely to be portable between different job submission methods:

    [[foo]]
         [[[job submission]]]
             method = loadleveler
        [[[directives]]]
             class = General

It would be better to have something like this:

    [[foo]]
         [[[job submission]]]
             method = loadleveler
             #method = pbs
        [[[loadleveler directives]]]
             class = General
        [[[pbs directives]]]
             # ...

[matthewrmshin]

(@dpmatthews and I discussed something like this a few weeks ago, but you have beaten us to raise an issue!)

Is this a desire to allow a suite to be portable to different batch systems within a site, or is it to allow a suite to be portable to different sites? The proposed solution will work for the former, but will not be enough for the latter because:

• Job hosts, environment and/or scripts on different sites are likely to be different.
• Even if the same batch system is used on 2 different sites, resource requirement for the same job may be very different due to different hardware and/or architecture.
• Options such as Loadleveler's class or PBS's -q are likely to be different on different sites.

[hjoliver]

I was thinking within a site, if you happen to have access to several batch schedulers. However, if we can think of a solution that would make it easier to switch between sites, that would be great. I guess that currently requires heavy use of Jinja2 switching and probably include-files.

Maybe we really need something like this?:

[runtime]
   [[foo]]
      script = run-foo
      [[[SITE-A]]]
           # job submission method, directives, hosting, ... for SITE-A
      [[[SITE-B]]]
           # etc.   

That would be a difficult backward compatibility problem, however.

[matthewrmshin]

That would be a difficult backward compatibility problem, however.

We can continue to support the current interface like this:

[runtime]
    [[task1]]
        # Continue to allow non-site-specific stuffs?
        script = command
        [[[job submission]]]
            method = background
        [[[SITE-A]]]
            script = command-only-at-site-a
            [[[[job submission]]]]
                method = pbs
            [[[[remote]]]]
                host = hpc-at-site-a
        [[[SITE-B]]]
            # ... and so on

[trwhitcomb]

With the new parsing that came online a while back allowing suite.rc snippets, there's a really (I think) clean solution to this by refactoring out the specific platform specific information into a separate file. I've been using this to move between systems and it's worked well. Here's what I did:

1. Define the suite as normal
2. Then, remove the [[[job submission]]] blocks out to a separate file. I chose to create an inc directory and then have the platform-specific suite.rc snippets in there.
3. Link the appropriate file to a platform.rc file which is then included in the main suite.rc

For me, I didn't need to worry about switching the entire scheduler but I did need to modify queue information as well as the specific resource requests.

[hjoliver]

[meeting]

• holding this pending further discussions on suite portability methods

[matthewrmshin]

Suite portability related to #1829.

[hjoliver]

Superceded by #2415