Optional quda runtime configuration using a file

[mathiaswagner]

Just wondering: We currently do have some options in QUDA that are not exposed in the interface, e.g. using reconstruction within MILC. Either we can hard code them as they are right now or expose them to MILC which will probably result in them just being hard-coded in the MILC code. That clearly is no real improvement.

Other option would be to have a compile flag (-D<something>). But using a compile time flag for setting essentially a runtime decision in QUDA is also not really desirable and messes up the build options for QUDA.

One way to avoid this would be an optional QUDA runtime configuration using a file that could be specified using an environment variable (like we do for the tunecache) or using environment variables to control QUDA runtime decision that are not exposed to the end user (either by QUDA or the calling application).

We would keep the currently hard-coded options as default.

A couple of questions:

• Is this feasible ?
• Is there a need for this ?
• Is it desirable or is this just a hack to bypass the interface ?
• Do we want this?

[maddyscientist]

I really like this idea, and it would be applicable for more than just MILC (e.g., BQCD, TIFR, etc.). It would have the additional advantage of reducing the amount of code required in the host application since the parameter setting is removed and would unify the setting of parameters. It would even be advantageous for Chroma, since it would allow for setting of new parameters in QUDA without waiting for changes to be propagated into Chroma.

Thoughts on an XML parameter file driven approach?

[mathiaswagner]

I didn't really think about details yet. I think there are a few requirements:

• Available (license-compatible) parser
• User-understandable setting of parameters
• we might want a hierarchical structure for the file
• Tools to check syntax before starting a run(?)

I actually like plist a lot (Mac OS property lists) when writing python code but am not sure about the parser for cpp. I only use them in Python and that comes with batteries included.

Maybe we have some feedback from others on that???

[mathiaswagner]

One more thing that came to my mind: Apart from setting it using an environment variable we might also allow the calling application to specify the configuration file using the interface? Maybe as an optional argument to init?

[rbabich]

I think the file should be human-readable. XML fails that test (at least for this human :-) ).

My vote would go to YAML: https://en.wikipedia.org/wiki/YAML

LibYAML is MIT-licensed: http://pyyaml.org/wiki/LibYAML

... but until we need something more complicated, a home-grown parser might suffice. I just uploaded this one, in case it's useful: https://gist.github.com/rbabich/9359c850aff56e17463f

[rbabich]

As a historical note, the reason QUDA_RESOURCE_PATH has the name it does (rather than something more obvious like "QUDA_TUNECACHE_PATH") is because we were envisioning that the directory might be used for additional purposes such as this one.

I like this idea, but I think it's important to make it as idiot-proof as possible. If a parameter is specified both via the interface and in an input file, I'm not sure which should take precedence, but a prominent warning should be printed either way. It's a little scary to think about someone doing a full production run at the wrong mass or using the wrong boundary conditions because of the presence or lack of an environment variable.

[mathiaswagner]

Agreed.

I thought more about passing things like whether to use reconstruction for the links and stuff like that. I.e. things that are not exposed in the interface or even if we expose them in the interface the calling application will have not way to expose them to the user (Imagine MILC would have to expose options to reconstruct links, like I said above).

Things that can and should be set from the calling application (like the mass) should not be exposed.

[alexstrel]

like the idea too: some parameters are attributed to QUDA library and cannot be so transparent in the application code (e.g., hard-coded reliable updates delta in the MILC interface, while MILC itself, if I'm not mistaken, does not support mixed precision optimization at all)

[mathiaswagner]

I am more into the XML direction than YAML but in the end the syntax of the input file matters only in terms of readability and ease of use (not creating errors in the file). But YAML looks promising.

Plist still might be useful but I could not find an parser that is easy to include so far (python comes with it). Only benefit would be if we want a python script to act as a parser / checker before (which can run basically anywhere) before noting that a large job crashed because of a syntax error in the input file.

But I guess this can also be done for YAML and I will throw in another option: JSON and the parser available at https://github.com/kazuho/picojson (Friendly license and not having an external dependency is nice).

But finally out implementation should not rely on a specific parser, so we can just get started with this.

I would prefer to have that infrastructure working at the interface level and just set parameters that are set within the interface. The fact that some things are set e.g. in milc_interface.cpp (reconstruction options, reliable_delta) might require some more work though.

[mathiaswagner]

One thing we need to evaluate first: Developer feedback seems positive but will users make use of it? Shall we ask them for feedback on this? Maybe @bjoo , @detar , @stevengottlieb , @azrael417 can comment from a user's perspective?

[bjoo]

HI All, This sounds like a good idea. With Chroma, I tried to leave all the myriad precision present in the XML files Chroma used resulting in very complicated interfacing classes.

One thing to consider is what your solvers are since now there will be many solvers with many different sets of parameters. E.g. Regular BiCGStab, GCR with inner DD solver, FGMRES DR, Multi-Grid, and so the space of these things will grow, especially as one starts using solver as preconditioners and smoothers inside other solvers.

Further it is possible that I may need a variety of solvers in my running (e.g. GCR + DD for 2 flavor and hasenbusch, Multi-Shift CG for single flavor, BiCGStab for whatever reason, an eigensolver maybe etc etc). So the file should allow to customize many QUDA solvers independently. A structured approach is good for this.

One possibility may be to have named solver configurations in the file e.g.:

Configuration Name = CG_SolverBalint1: <options for a particular solver instantiation, re precisions etc etc etc>

Then the interfacing code from Chroma could just refer to CG_SolverBalint1 (ie the Chroma XML could just hold this name in a tag). For housekeeping purposes, Chroma should (or maybe shouldn’t) be able to interrogate a configuration/change like this (e.g change quark masses, or algorithmic but not precision related parameters) It may very well make the Chroma-QUDA interfaces simpler and more stable over time as Mike points out, but of course the script we will use to write the new solver config file will need to evolve :)

Best, B

On Aug 3, 2015, at 4:38 PM, Mathias Wagner notifications@github.com wrote:

One thing we need to evaluate first: Developer feedback seems positive but will users make use of it? Shall we ask them for feedback on this? Maybe @bjoo , @detar , @stevengottlieb , @azrael417 can comment from a user's perspective?

— Reply to this email directly or view it on GitHub.

---

Dr Balint Joo High Performance Computational Scientist Jefferson Lab 12000 Jefferson Ave, Suite 3, MS 12B2, Room F217, Newport News, VA 23606, USA Tel: +1-757-269-5339, Fax: +1-757-269-5427

email: bjoo@jlab.org

[AlexVaq]

For us (ETMC) I think it would be very useful. I think it is easier to handle than the command line, and people here would appreciate it.

El 3/8/2015, a las 22:38, Mathias Wagner notifications@github.com escribió:

One thing we need to evaluate first: Developer feedback seems positive but will users make use of it? Shall we ask them for feedback on this? Maybe @bjoo , @detar , @stevengottlieb , @azrael417 can comment from a user's perspective?

— Reply to this email directly or view it on GitHub.