Support scatter/gather runs with metrics

[ConnectedSystems]

This PR finally closes #46 and #48

pinging @KRNA01 and @Rosejoycrocker

Still have to manually sum/average the results (see last line in the example below) but I think (?) this is acceptable.

Would either of you like to volunteer to do a review? :smile:

ai.runToDisk(sample_table, sampled_values=true, nreps=n_reps, ...
    file_prefix='./example_multirun', batch_size=4);

% Gather results, applying a metric to each result set.
% Y will be a cell array of result structs (one for each batch)
Y = ai.gatherResults('./example_multirun', {@coralTaxaCover, ...
                                            @coralEvenness, ...
                                            @coralSpeciesCover, ...
                                            @shelterVolume});

% Use `concatMetrics` to combine all runs together.
% This function should also work with a single result set (e.g., runs from `ai.run()`)

% Total coral cover
TC = concatMetrics(Y, "coralTaxaCover.total_cover");

% Coral cover per species
covs = concatMetrics(Y, "coralSpeciesCover");

% Collect coral evenness
E = concatMetrics(Y, "coralEvenness");

% Collect juvenile corals (< 5 cm diameter)
BC = concatMetrics(Y, "coralTaxaCover.juveniles");

% Collect coral shelter volume per ha
SV_per_ha = concatMetrics(Y, "shelterVolume");

% Calculate average total cover over all scenarios/replicates
mean(TC, [3,4])

Indicative plots from an example batch run with 8 scenarios, 3 replicates (24 simulations in total)

[KRNA01]

Thank you @ConnectedSystems, would be happy to review. More metrics are still to come (RCI, RTI and RFI), but these will largely be derived from these basic metrics, so can be in a separate PR. Pinging @ryanheneghan here, as we're working on this jointly.

PS: now that more co-developers are onboard, should we review the 'species' definition in the model to avoid confusion? 'Groups_and_sizes' would be the most descriptive (albeit clumsy) variable name for the 36 elements of the coral dynamics. Happy to kick this can down the road given we're on a sprint .. again

[ConnectedSystems]

Hi @KRNA01

I saw some recent changes to shelterVolume have happened. Please note that this PR includes some adjustments to allow use of current metrics with datasets that hold > 1 simulations. I'd like to see this merged in before the differences become hard to resolve.

should we review the 'species' definition in the model to avoid confusion? 'Groups_and_sizes' would be the most descriptive (albeit clumsy) variable name for the 36 elements of the coral dynamics.

I say we leave this to later, but:

• "grouped_taxa"
• "taxa_and_size"

Taxa is probably incorrect here too, happy to go with "group_and_sizes"

[KRNA01]

Great – yes, agree. K

From: Takuya Iwanaga @.> Sent: Monday, 24 January 2022 8:58 AM To: open-AIMS/ADRIA_repo @.> Cc: Ken Anthony @.>; Mention @.> Subject: Re: [open-AIMS/ADRIA_repo] Support scatter/gather runs with metrics (PR #66)

Hi @KRNA01https://github.com/KRNA01

I saw some recent changes to shelterVolume have happened. Please note that this PR includes some adjustments to allow use of current metrics with datasets that hold > 1 simulations. I'd like to see this merged in before the differences become hard to resolve.

should we review the 'species' definition in the model to avoid confusion? 'Groups_and_sizes' would be the most descriptive (albeit clumsy) variable name for the 36 elements of the coral dynamics.

I say we leave this to later, but:

• "grouped_taxa"
• "taxa_and_size"

Taxa is probably incorrect here too, happy to go with "group_and_sizes"

— Reply to this email directly, view it on GitHubhttps://github.com/open-AIMS/ADRIA_repo/pull/66#issuecomment-1019585106, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AVYCFW7LDWN4FRARBUCDQFTUXSBXBANCNFSM5MMBASQQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.**@.>>

[KRNA01]

PS: key changes to shelterVolume (from memory) was a math fix. Tested this for a single_scenario, and was comfortable with what I saw

[KRNA01]

@ConnectedSystems, I see that you want to calculate means (for TC) across all simulations. Could we also capture results of all simulations simulations somewhere, please? This is useful for some analyses where we compare the performance of the different metrics across interventions and intervention assumptions (e.g. violin plots). The original 5-dimensional matrix output was clumsy but useful.

[KRNA01]

@ConnectedSystems, happy for you to merge if these are things we can sort out later

[ConnectedSystems]

@ConnectedSystems, I see that you want to calculate means (for TC) across all simulations. Could we also capture results of all simulations simulations somewhere, please? This is useful for some analyses where we compare the performance of the different metrics across interventions and intervention assumptions (e.g. violin plots). The original 5-dimensional matrix output was clumsy but useful.

Yes, if you recall these metrics are applied after raw results are collected. When running in batch mode, those get stored directly to files. In the direct single/multi run modes you could save the results to file before collecting these metric values.

Will merge this in now.