[FEATURE]Add Schema Generate API

[YANG-DB]

Is your feature request related to a problem?

A user would like to generate the physical representation of the schema as its manifested in the database -

• component template mapping
• index template mapping

What solution would you like?

As part of the user-flow the domain users would like to generate the physical mapping for their schema representing the templates from which the indices (schema instances) would be created and used.

API

GET _plugins/catalog/schema/{id}/generate?version=1.0&prefix=ss4o&suffix=*

This generate API takes all the schematic components belonging to schema id defined in the catalog repository and generates the appropriate physical mappings representation of the schema.

We are using the Naming-convention that will automatically create the names for the index mapping template according to their type.

Possible parameters

• version - mandatory semantic version of the schema

• type - optional schematic category type (or list of types) to generate (default is `_all)

• components - optional schematic category component (or list of components) to generate (default is `_all)

• dataset - optional field can contain anything that classify the source of the data - such as nginx (default is *)

• namespace - optional user defined namespace - useful to allow grouping of data such as production grade, geography classification (default is *)

• prefix - optional prefix name identifies the context of the entire naming (default is ss4o)

• suffix - optional suffix name identifies the context of the specific index pattern - may be date or sequence (default is *)

Example 1:

This following call GET _plugins/catalog/schema/observability/generate?version=1.0&prefix=ss4o will generate the following templates:

Component Templates

• ss4o_1.0_http_component --> corresponds to this
• ss4o_1.0_container_component --> corresponds to this
• ss4o_1.0_cloud_component --> corresponds to this
• ss4o_1.0_communication_component --> corresponds to this
• ... rest of the components ...

Index Templates

• ss4o_1.0_logs-*-* --> corresponds to this
• ss4o_1.0_traces-*-* --> corresponds to this
• ss4o_1.0_metrics-*-* --> corresponds to this
• ss4o_1.0_services-*-* --> corresponds to this

---

Example 2:

This following call GET _plugins/catalog/schema/observability/generate?version=1.0&prefix=ss4o&type=[traces,services]&namespace=us will generate the following templates:

Component Templates All the components that are needed to the generate the types described in the type list:

• ss4o_1.0_http_component--> corresponds to this
• ss4o_1.0_container_component --> corresponds to this
• ss4o_1.0_cloud_component --> corresponds to this
• ss4o_1.0_communication_component --> corresponds to this
• ...

Index Templates

• ss4o_1.0_logs-*-us --> corresponds to this
• ss4o_1.0_traces-*-us --> corresponds to this

---

Example 3:

This following call GET _plugins/catalog/schema/observability/generate?version=1.0&prefix=ss4o&type=logs&components=[http,communication] will generate the following templates:

Component Templates All the components that are needed to the generate the types described in the type list:

• ss4o_1.0_http_component --> corresponds to this
• ss4o_1.0_communication_component --> corresponds to this
• ...

Index Templates

• ss4o_1.0_logs-*-* --> corresponds to this

---

Metadata -embedded

As part of the creation of the mapping template (both index and component) the schema metadata that specifies the logical schema origin and version is embedded into the mapping template structure under the _meta field:

  "template": {
    "mappings": {
      "_meta": {
        "version": "1.0.0",
        "catalog": "observability",
        "type": "logs",
        "component": "http"
      },

See examples :

• logs meta info

• http meta info

  Naming-convention

  Naming Index patterns will follow the next naming structure prefix_{version}_{type}-{dataset}-{namespace}_suffix

version - mandatory semantic version of the schema (1.0 / 1.2 ) type - indicated the observability high level types "logs", "metrics", "traces" (this is taken from the category within the catalog) dataset - a field can contain anything that classify the source of the data - such as nginx - (If none specified * will be used). namespace - a user defined namespace - useful to allow grouping of data such as production grade, geography classification

Prefix - optional prefix name identifies the context of the entire naming Suffix - optional suffix name identifies the context of the specific index pattern (may be date or sequence )

The sso_1.0_{type}-{dataset}-{namespace} Pattern address the capability of differentiation of similar information structure to different indices accordingly to customer strategy.

For example a customer may want to route the nginx logs from two geographical areas into two different indices:

• ss4o_1.0_logs-nginx-us
• ss4o_1.0_logs-nginx-eu

This type of distinction also allows for creation of crosscutting queries by setting the next index query pattern sso_1.0_logs-nginx-* or by using a geographic based crosscutting query sso_1.0_logs-*-eu.

Do you have any additional context?

• https://github.com/opensearch-project/opensearch-catalog/blob/main/docs/schema/observability/Naming-convention.md
• https://github.com/opensearch-project/opensearch-catalog/tree/main/docs/schema/API
• https://github.com/opensearch-project/opensearch-catalog/tree/main/schema/observability

[Swiddis]

Some initial questions:

• Why is it named generate? My understanding is that we want to retrieve an existing schema, not make a new one.
• How do we retrieve the latest version?
• What does it mean to get observability version 2.0.0? Or to specify a version of any collection of types?
  • That every returned template will be version 2.0.0? Does that imply that every template must be updated at once, even if only one of them changes?
  • Related, we currently have no built-in mechanism to specify the version of component template dependencies for index templates. Is that stable?
• The suffix parameter may be a date or sequence — What does “sequence” mean here? Why these types?
  • Is defaulting to * backwards-compatible with existing integration behavior that has a ss4o_*-*-* format?
• Do we have a way of distinguishing component and index templates before calling GET? Are there any templates that can be both? If we get a template without specifying, how do we tell which it is?

[YANG-DB]

• generate - stands for generating the physical mapping from the logical schema

• retrieval - is called using the GET /schema/Observability/ API in case multiple versions exist the result will be a dictionary of version:schema, this will retrieve the logical structure only (catalog.json)

• The actual instances created from the logical schema (using the mapping) are fetched using this api /schema/{id}/instances

  • Do you think a dedicated API for GET /schema/Observability/_mapping is needed ?

• As you mention correctly currently we assume that all components within a type (logs/traces) are updated within a single version (http, communications, cloud ..)

• suffixes might be sequence number as used by data-prepper for example to partition data based on time, date is used for jaeger to achieve the same

• ss4o_*-*-* is the most generic pattern that will catch all index names that start with these characters - it may be too wide for some users so we allow the agility for setting the different naming parts

• the schema structure is well defined within the catalog.json which details all the schema inner hierarchy

Once a index / component template was created - we can distinguish its source and version using the _meta details embedded within the mapping file itself:

• logs meta info
• http meta info