All feb 2023 improvements so far

[kelvinburke]

Can give greater detail on Monday, but this is ready to merge - all backwards compatible and doesn't change results

!! sciris will need to be updated on the server and in local environments: to Version 2.0.2 (2022-10-22) at least. pip install sciris --upgrade

[kelvinburke]

On loading the scenario tab, the project gets the default coverage of all the programs for all times for all combinations of parsets and progsets. This is sped up by not looping over t values, but supplying the t values as an array.

This gave different results compared with inputting the t values one by one.

This was caused by two issues which have now been fixed:

• The popsize interpolation would apply the exponential curve, even if it wasn't needed yet if only one t value was input. (Note that this fix likely fixes weird issues with the reconciliation page!) UPDATE, this is now a 2.12.x change as it changes results slightly.
• When there are multiple cost-coverage curves, the model interpolates between these. This was being smoothed if an array of t values was input - leading to inconsistent results. Now, no smoothing is done, but still being interpolated.

[kelvinburke]

Overall major updates are:

Scenarios page loads quicker because of the fixes above (one of which is a 2.12.x change because although it fixes a problem, it changes results)

Introduction of absconstraints and proporigconstraints: When calling P.optimize the suggested is to provide proporigconstraints since they work well on the FE too: FE only uses proporigconstraints and will delete constraints and absconstraints if an Optim is open and Save is clicked on the FE The optimization will use the stricter of the 3 when starting an optimization and turning those into absconstraints that are used throughtout the optimization

initpeople works consistently if you follow the instructions in analyses/general/system_tests/check_initpeople_consistency.py and get the initpeople a year before you make any changes due to smoothing

Speedups in the calcalation of parameters, especially for the parameters from a budget scenario

These two combined means optimizations take approximately 25% of the time!

Fixed the FE overwriting changes to a project when an optimization finishes

Optimizations now give two scenarios "Baseline" and "Optimization baseline" if they are different. The latter is rescaled and constrained.

MC sampling at the start of an optimization is now improved and given as a tuple mc: (baselines, randoms, progbaselines) counts the number of optimizations to start with each of: baselines start from the origbudget (rescaled and constrained) randoms start from a randomized (constrained) budget progbaselines start with all money in a single program

a multi chain (nchains > 1) optimization is no longer recommended, instead use mc nblocks > 1 doesn't seem to do much either, but will quit early if making no improvements

nblocks > 1 runs mc on the first block, but not on the subsequent. It takes a number of the best runs and continues those.

P.optimize has new parameters: parallel which replaces multi. parallel is whether you want the mc sampled starting points to run in parallel ncpus is the number of threads you want to use blockiters is no longer used, now only use maxiters

The suggested inputs to P.optimize are: parsetname, progsetname, objectives, proporigconstraints, maxtime, ncpus, randseed

So maxtime will control how much mc sampling will be done, the 3 cases are maxtime <= 60, maxtime > 60, maxtime = None. Check the default settings in P.optimize and optimize in tasks.py

randseed works more consistently with mc sampling - same results unless we timeout at different points

!! sciris will need to be updated on the server and in local environments: to Version 2.0.2 (2022-10-22) at least. pip install sciris --upgrade