[RFC] Options for test registries, within DESCQA

[nsevilla]

This request for comments pretends to poll for some ideas to incorporate a registry of executed tests for the DESC Test Framework, under development as a branch of DESCQA (TBC).

I latched on to @yymao's idea of creating a registry of tests for DESCQA, that would contemplate also tests not executed with the descqarun pipeline, but instead would incorporate results and provenance information in a standardized way, and output them on the same footing as the descaqrun tests.

I see 2+1 options:

• Register them in the same way as descqarun tests, so we would have to create some descqarun like configuration files for these tests, to be incorporated in the outputs. (I am inclined to pursuing this option right now)
• Build something 'outside' descqarun execution, with an independent registry which would incorporate/read descqaruns own registry at the end.
• Something else I did not think of, possibly because I am misunderstanding DESCQA workings.

[yymao]

I think Option 1 would work well. We can include a flag in the yaml config to indicate whether the config is for an external test or a test can be run with descqarun.

We actually use a very similar system in GCRCatalogs. See https://github.com/LSSTDESC/gcr-catalogs/blob/master/CONTRIBUTING.md#preparing-a-catalog-config. In particular, we have a reserved keyword is_pseudo_entry, which when set to true, it indicates the config is not intended to be loaded via GCR, and there is no reader associated with it.

[nsevilla]

I am closing this as @yymao's option is quite straightforward. We can follow up on any implementation details or problems in the descqa repo, once this has been successfully merged.