Better system for running examples #50

Closed podhrmic closed 3 years ago

In GitLab by @podhrmic on Sep 2, 2020, 10:56

Right now, we have two shell scripts that let the user to select a predefined example to run. As the number of examples increases, we should have a better way to select between them (better than manually adding each example to the shell script selector - that seems tedious and error prone).

Some options:

use different conf.toml for each example. The conf file would contain which example to run, and some other simulation params (tspan etc.). We would have conf_f16_simple.toml, conf_f16-gcas.toml etc.
use a python script (instead of shell script), and utilize command line argument (with argparse or some other handy library)

@bauer-matthews @EthanJamesLewthoughts?

In GitLab by @bauer-matthews on Sep 2, 2020, 11:12

I think we already have a different TOML config for each of the examples we currently support. We can push more of the simulation params into system level the TOML files. Although tspan in particular doesn't seem like something that belongs there. Since its more of a generic parameter that pertains to simulations across all models (its not a property of the model itself).

An easy thing to do would be to just search the examples folder for system level TOML files and present those as options to the user.

In GitLab by @podhrmic on Sep 2, 2020, 11:31

Do I understand it right that the additional params should be a part of lets say f16_simple_config.toml?

For example:

# Simulation configuration
[simulation]
  tspan = [0.0, 5.0]
  show_status = true # show progress during simulation

[display]
  plot = true # plot results on screen
  render = false # render a 3D animation
  fg = false # display simulated trajectory in FlightGear

That definitely makes sense, and lets us to keep all the settings related to a single system in one place.

Now, is there a good way to handle default values for optional entries in the conf file? Let's say all the parameters above are optional - then I would like to have something better than:


# Get default simulation settings
show_status = True
tspan = [0, 10.0]
analyze = False

# Get simulation specific params
if 'simulation' in my_conf.config_dict.keys():
    print("Using simulation settigs from TOML file")
    sim_conf = my_conf.config_dict['simulation']
    show_status = sim_conf['show_status']
    tspan = sim_conf['tspan']

In GitLab by @EthanJamesLewon Sep 2, 2020, 11:38

@podhrmic

That definitely makes sense, and lets us to keep all the settings related to a single system in one place.

This seems suboptimal. A system can be used in a variety of ways, and to specify the simulation/analysis tasks alongside the system description prohibits other uses (or complicates configuration) of the system. The old f16 repo had such a separation between the model and its demo configuration.

In GitLab by @bauer-matthews on Sep 2, 2020, 11:41

Looking at the params in more detail, I agree with @elew. I don't think simulation parameters are a component on the model. And the display parameters are not generic enough across systems (i.e. fg is only relevant to certain aircraft models).

Maybe its worth having a quick check-in.

In GitLab by @podhrmic on Sep 2, 2020, 11:42

EDIT: I think using the dictionary as myVal = dict.get('someKey', someDefaultValue) is the way to go.

@EthanJamesLewwhat would be your preferred solution? What about a separate & optional conf.toml with the simulation / visualization parameters?

In GitLab by @EthanJamesLewon Sep 2, 2020, 11:44

What about a separate & optional conf.toml with the simulation / visualization parameters?

I think that this is reasonable. Having a separate toml file for this produces an abstraction for a job description, something that can understood and operated on.

In GitLab by @podhrmic on Sep 2, 2020, 11:47

Prior our checkin - here are some use-cases:

The user wants to run the model with a different time span, and with different initial conditions. I expect this to be the case quite often.
The user wants to run the model in some sort of analysis framework (could be the control system "fuzzer") - then besides the parameters changed in 1) it is also necessary to specify some steps after the simulation (calculate error etc.).

In GitLab by @bauer-matthews on Sep 2, 2020, 11:48

So what all would be in a job configuration? A tspan and a dictionary for example specific options like fg?

In GitLab by @bauer-matthews on Sep 2, 2020, 11:49

So we would be building an orchestration layer on top on the core simulation engine.

In GitLab by @podhrmic on Sep 2, 2020, 11:55

So what all would be in a job configuration? A tspan and a dictionary for example specific options like fg?

Single sim specific: tspan, show_status (show progress during simulation), plot, initial_conditions
Example specific: fg, render
Control system test specific: max_iterations (for now), and a way to set initial_conditions for each iteration

So we would be building an orchestration layer on top on the core simulation engine.

That is my understanding. Although this layer can be pretty simple.

In GitLab by @podhrmic on Sep 2, 2020, 12:11

Actually, thinking about it more, the orchestration framework can be pretty lightweight - what I really need is to have a convenient way to store all the job specific params. And a separate conf.toml file does the job perfectly.

In GitLab by @podhrmic on Sep 15, 2020, 16:31

mentioned in merge request !26

In GitLab by @podhrmic on Sep 19, 2020, 17:17

mentioned in merge request !40

In GitLab by @podhrmic on Sep 22, 2020, 10:13

mentioned in commit eb48356e91d5b408d15f317114ef160926366307

GaloisInc / csaf

Better system for running examples #50