Critical: Clean up configuration

[mturilli]

Troy leverages three configuration methods for a session with a unified configuration file + inline configuration in the application code. We need to:

• Unify the configuration model across plugins, especially across plugin for the same functionality.
• Consolidate the configuration into a single file.
• Minimize the mandatory configuration keys.
• Discriminate between mandatory/optional configuration keys.
• Define sensible default values for the optional keys.
• Raise appropriate exceptions.

Here an analysis of what we have so to kickstart the discussion.

Configuration files (~/.troy.cfg)

• I would keep the configuration file only for the information that is not application/run specific.
• We can discuss how to manage this file: Sagapilot approach improved by a system of local caching?

[overlay_translator_default] (Move to the application code) pilot_size -> Optional (default: #tasks = #cores)

[overlay_scheduler_round_robin] (Move to the application code) resources -> Mandatory

[planner_concurrent] (Move to the application code) concurrency -> Optional (default: 100)

[workload_dispatcher_bigjob_pilot] (Change name: drop 'pilot'. Move to the application code) coordination_url -> Mandatory

[overlay_provisioner_bigjob_pilot] (Change name: drop 'pilot'. Move to the application code) coordination_url -> Mandatory

[workload_dispatcher_sinon] (Change name: sagapilot. Move to the application code) coordination_url -> Mandatory

[overlay_provisioner_sinon] (Change name: saga pilot. Move to the application code) coordination_url -> Mandatory

[compute:india] endpoint -> Mandatory sinon_id -> (Drop: We should not ask this to the user) type -> (Drop: We should not ask this to the user) username -> (Move: This goes into/passed to the application) home -> ??

[gromacs_demo] (Move: This goes into the application code) pilot_backend -> Move-Mandatory bagsize -> Move-Mandatory steps -> Drop: Specific to mdrun, goes into the kernel description demo_id -> Move-Optional user -> Move-Mandatory mdrun -> Drop: This goes into the kernel description home -> ?? Do we have a sensible default in saga pilot/BJ? resources -> Move-Mandatory

Plugin configuration:

• I would like to drop the dictionary based configuration in favor of passing configuration parameters at instantiation time. The more I use that code, the less I find comfortable with it. It may work for owms but, again, it really feels clunky to me. So, from:

  session.user_cfg['overlay_scheduler_round_robin'] = { 'resources' -> Mandatory } session.user_cfg['planner_concurrent'] = { 'a' : 'b' -> ?? 'concurrency' -> Optional } session.user_cfg['overlay_translator_max_pilot_size'] = { 'pilot_size' -> Optional }

to:

overlay_translator = troy.OverlayTranslator(plugin='max_pilot_size')
overlay_scheduler = troy.OverlayScheduler(plugin='round_robin', resources = resources) 
planner = troy.Planner(plugin='Concurrent')

overlay_mgr = troy.OverlayManager(scheduler = overlay_scheduler, translator = overlay_translator)
workload_mgr = troy.WorkloadManager()