Adds a run configuration option to skip the calculation of measurement pairs which can be resource-intensive for certain runs. See #654.
The new option is variability.pair_metrics and is True by default to match the existing behaviour. Runs with the pair metrics disabled will not write a measurement_pairs.parquet file. Similarly, the arrow format equivalent is not written if requested. The maximum Vs/m metrics stored in the Source models are all set to 0.
Measurement pairs are also calculated when rendering the plots on the source detail page. For runs where the pair metrics are turned off, this step is now skipped. Some changes were required to the PipelineConfig class to allow the run config to be read and parsed, but not validate the inputs. Skipping input validation in this scenario is desirable as large runs can take a long time to validate while the glob expressions are resolved. The existing validation skip was insufficient as we need the variability.pair_metrics value to be parsed (i.e. the value must be a bool as dictated by the schema - without validating the values are left as strings).
Adds a run configuration option to skip the calculation of measurement pairs which can be resource-intensive for certain runs. See #654.
The new option is
variability.pair_metricsand isTrueby default to match the existing behaviour. Runs with the pair metrics disabled will not write ameasurement_pairs.parquetfile. Similarly, the arrow format equivalent is not written if requested. The maximum Vs/m metrics stored in theSourcemodels are all set to 0.Measurement pairs are also calculated when rendering the plots on the source detail page. For runs where the pair metrics are turned off, this step is now skipped. Some changes were required to the
PipelineConfigclass to allow the run config to be read and parsed, but not validate the inputs. Skipping input validation in this scenario is desirable as large runs can take a long time to validate while the glob expressions are resolved. The existing validation skip was insufficient as we need thevariability.pair_metricsvalue to be parsed (i.e. the value must be a bool as dictated by the schema - without validating the values are left as strings).Will update the changelog when tests pass.