Adds a structured_logs array property to the Log schema that allows implementers to provide well-defined structure log information for run (via RunLog.run_log) and task logs (via RunLog.task_logs[]).
The structured_logs property uses a custom schema of type StructuredLog that defines a model or schema (log_type) of a given version (log_type_version) that describe the shape of the log contents (logs). Optional properties comprise pointers to additional information, in particular a URL to the specification of the model or schema itself (spec_url) and a URL to human-readable implementation notes (doc_url) describing how to consume the log contents.
Valid values for log_type are defined in the enumLogType and their mapping to the log contents in logs should be sufficiently well described in the enum description such that server-side implementers can reasonably be expected to provide the logs in a uniform way, while client-side implementers can reasonably be expected to consume them as long as they provide support for the indicated log_type_version.
The enumLogType is nullable. Specifying null can be used to provide information on structured log formats that are not (yet) explicitly supported by WES. In such cases, it is strongly advised that at least one of spec_url or doc_url are provided to allow clients to interpret the log concents.
Open questions:
Given that the proposed mechanism may also be relevant for TES, the StructureLog and LogType schemas could also be defined in a separate repository where schemas are kept that can be reused across multiple APIs (perhaps governed by the Cloud WS or DaMaSC?).
While the logs property was defined in a deliberately promiscuous way (it can take a string, an object or an array of arbitrary values), it is not clear if the proposed schema is sufficient for supporting a wide range of external log format models and schemas.
The proposed code adds the Workflow Run RO-Crate profile collection as known/defined structured log formats. This is motivated by an important use case for the ELIXIR Cloud & AAI Driver Project. Moreover, mapping the Workflow Run RO-Crate profile collection should be easy, given that RO-Crates are JSON-LD documents that should be relatively trivial to return as a single string item. However, if adding a set of concrete log formats goes too far at this point, these can also be added at a later point.
OpenAPI 3.1 provides a way to annotate individual enum values via oneOf and const (see here). Adopting this would greatly improve upon the maintainability of the LogTypeenum. However, it would require to upgrade WES to OpenAPI 3.1.
Implementation notes:
structured_logsarray property to theLogschema that allows implementers to provide well-defined structure log information for run (viaRunLog.run_log) and task logs (viaRunLog.task_logs[]).structured_logsproperty uses a custom schema of typeStructuredLogthat defines a model or schema (log_type) of a given version (log_type_version) that describe the shape of the log contents (logs). Optional properties comprise pointers to additional information, in particular a URL to the specification of the model or schema itself (spec_url) and a URL to human-readable implementation notes (doc_url) describing how to consume the log contents.log_typeare defined in theenumLogTypeand their mapping to the log contents inlogsshould be sufficiently well described in theenumdescription such that server-side implementers can reasonably be expected to provide the logs in a uniform way, while client-side implementers can reasonably be expected to consume them as long as they provide support for the indicatedlog_type_version.enumLogTypeis nullable. Specifyingnullcan be used to provide information on structured log formats that are not (yet) explicitly supported by WES. In such cases, it is strongly advised that at least one ofspec_urlordoc_urlare provided to allow clients to interpret the log concents.Open questions:
StructureLogandLogTypeschemas could also be defined in a separate repository where schemas are kept that can be reused across multiple APIs (perhaps governed by the Cloud WS or DaMaSC?).logsproperty was defined in a deliberately promiscuous way (it can take a string, an object or an array of arbitrary values), it is not clear if the proposed schema is sufficient for supporting a wide range of external log format models and schemas.stringitem. However, if adding a set of concrete log formats goes too far at this point, these can also be added at a later point.enumvalues viaoneOfandconst(see here). Adopting this would greatly improve upon the maintainability of theLogTypeenum. However, it would require to upgrade WES to OpenAPI 3.1.Resolves #215