[RFC] Options for validation platforms for upcoming data

[nsevilla]

Summary Rubin will produce data sets in the form of Data Previews and Data Releases on which the SRV group would like to run a consistent set of tests and analyses for general characterization of the data, reporting on data related problems and regression purposes. We would like to address beforehand the most convenient way of building this software + analysis platform.

Description A couple of options are outlined below. This RFC will hold comments and further options.

• Prior to June 2021, the DESCQA platform was created to address several of the points in the Summary, as well as others specific to the simulations it was catering to. The job going forward would be to investigate scaling options if needed, reassess the necessary tests and analyses, investigate new visualization approaches, reassess the interfaces to data when the formats and access modes are decided.
• The Rubin project is developing the faro QA platform, it could be a complement or alternative.
• We could build something ad-hoc with the necessary tests, building on some of DESCQA experience, but also from existing software from DESC members devoted to these validation and characterization purposes.

[nsevilla]

During April-June, at the Computing WG DESC telecons, these possible approaches were presented. Further discussions with DESCQA users and developers made Javi and I think DESCQA was a good option, due to the extent of its development and being tailored for DESC. However, it was pointed out that it would be useful to have the discussion and decision steps laid out here to settle the decision and future reference.

[nsevilla]

• DESCQA: pros: very extensive and recent development. Developers still active in DESC. Code tailored for DESC. Cons: needs evaluation of scalability, development for running on non-NERSC platforms, reassess visualization options.
• faro: pros: it is being actively developed by the Rubin project, large development effort being put forward right now in a very timely manner. Cons: tailored to produce scalar metrics on calexps mainly (for now).
• Own code: pros: we can decide from the ground up what development approaches are more suitable for the characterization and regression analyses of forthcoming DPs and DRs. Cons: large development effort needed.
• The above could be mixed into a new DESC SRV platform.

[nsevilla]

I am going to close this RFC, as it has been superseded by issue #9 .

[nsevilla]

Reopening the RFC, to address the possibility of incorporating analysis_tools as part of the test framework.

[nsevilla]

During the 2022 August meetings (DESC and PCW), several people expressed that the use of Rubin's analysis_tools as a part/totality of our testing strategy would be good to:

• Efficiently give feedback to Rubin.
• Collaborate on the development process.
• Take more easily our tests and results upstream of DESC.

At the September 13th telecon , we reviewed this and several concerns were expressed, mainly:

• The need to settle on an option ASAP, instead of switching platforms every time new code was made available from whatever source. It was expressed at the meeting that this was a recent development at the project that was more suited to our needs potentially than previous iterations (eg faro) and that most probably analysis_tools was here to stay.
• The need to keep the software we use under our own control, to allow for rapid changes and adaptations to our needs.

Exploring analysis_tools capabilities was seen as a worthy task for Sprint Week.

[nsevilla]

Some conclusions and materials from the sprint by @psferguson @patricialarsen can be found at https://github.com/LSSTDESC/SRV-planning/tree/main/analysis_tools_guides

[nsevilla]

We are currently continuing work on our test framework using DESCQA as our baseline. There are several reasons for this:

• When this option was adopted (late 2022), the most important alternative, switching to analysis_tools, was not 100% viable due to the latter being in a ramp up of intensive development.
• However, we have seen that it is possible to adapt analysis_tools tests to DESCQA. We should keep the option open, for the reasons commented above (interoperability with Rubin). Will keep the RFC open due to this latter point.