Monte Carlo Optimization Improvements

[jasonlharvey]

Expand on the MC/PSO prototype from (#64):

• [x] Separate the PSO optimizer from the MC manager.
• [x] Create a generic interface between the MC manager and optimization schemes implemented as derived classes.
• [x] Implement a gradient descent optimizer.
• [x] Embed all Trick calls (MC functions, etc.) in an included macro file (similar to but separate from SimCompatibility) for ease of porting outside of Trick
• [x] Add ability to apply simple constraints of MC input variables relative to each other, e.g. input_A = 0.5 * input_B, etc. We could maybe implement simple ones like multiply & addition, and provide an interface for users to supply their own custom functions.
• [x] Cleanup the code & unit test
• [x] Update the wiki: https://github.com/nasa/gunns/wiki/Optimization
• [x] Fix parallelized runs from pso.py: running in parallel with pso.py, all the run costs remain zero and no useful state is found. The pso_state.csv results in all zero costs. Function propagateSwarm() is never getting called when running from pso.py, but it works just fine in serial mode from RUN_mc/input.py.

Further follow-on work:

• Trick is improving their MC to be able to call slave runs in batches - when they do that, then we should replace our pso.py Python wrapper and run all the iterations within a single Trick MC run.
• Implement other optimization schemes: adaptive genetic, ameoba (see Trick's example), etc.
• Provide an interface for users to select from alternate cost functions, or provide their own cost function

[jasonlharvey]

Some notes from the initial prototype: some of this info is obsolete now, but some should be preserved in the user's guide documentation for this stuff:

More ideas for improvements:

• ...

Other notes:

• number of MC slaves corresponds to # parallel runs (CPU's) in the compute farm, not the PSO swarm size
• note that for very short slave runs (small models over short simulated time length), the serial approach is actually faster than the parallel, multi-core runs (pso.py python wrapper), because of the overhead of starting/stopping the main trick sim for each epoch. However for larger models over more realistically long slave runs, parallel is faster.
• We can't use Trick for multiple epochs when using parallel, multi-core slave runs, because Trick provides no way for MC users to run the slave runs in batches. For example, if we have a swarm size of 30, running on 10 cores, we want to kick off the 30 runs of the first epoch, get all of those 30 results back, and then update the optimizer, before starting the next batch of 30 for the next epoch. Trick can't do that; rather it just runs all slave runs (#particles #epochs) as fast as it can, and results come back from each run in a random order. This is why we use the pso.py Python wrapper for parallel runs - it runs each PSO epoch (swarm iteration) as a separate Trick MC run (#particles 1 epoch), and then communicates between iterations via saved swarm states in files. The overhead of starting & stopping separate Trick sims between each iteration slows the whole process down, but it can still be faster than running serially, for long slave runs. UPDATE: Trick is implementing a way for us to run the slaves in groups, all in a single Trick master sim run, and we will upgrade our stuff to use that when it's ready.

PSO advice from Chip Birge/Intuitive Machines:

• recommend lookup 'Pareto front', multi-objective optimization
• recommend use 24-30 particles
• try initial particle states in the corners, or maybe all together in a single corner, etc. (we have this as an option, and it works)
• after a main run (multiple PSO epochs), use the saved state of the run as input to another run, with lower velocity limit because you know you're closer to the global min (we've tried it, it works)