LogsDB - Third-party Integration Testing

[salvatore-campagna]

Description

As part of the effort to test integration of LogsDB we would like to run third-party integration testing. When it comes to integrations, customers are allowed to override some of the LogsDB settings. We need to make sure failures takes place before indexing can start (fail fast) so to avoid documents being dropped at indexing time. Customers can integrate directly using LogsDB by means of their own templates and custom templates or by overriding settings by means of logs@custom component template. It should not be possible for a user to end up in a situation where index settings and/or mappings are conflicting or have forbidden values or forbidden combinations of values.

• [x] Update the losg@custom template to override index sort settings using a set of fields different from the default (host.name, @timestamp). Create a new datastream making use of the default logging templates and the logs@custom template. Make sure the backing index for the datastream has the correct index sort settings according to what is overridden by logs@custom.
• [x] Update the logs@custom template to override the index source mode to be “stored” (rather than the default “synthetic”). Make sure the logs@custom template update request fails. Using synthetic source is required for LogsDB and trying to disable it by overriding the source mode setting should be validated and fail before the template is used.
• [x] Update the logs@custom template to override the index codec using “best_speed” (overriding the default “best_compression”). Create a new datastream making use of the default logging templates and the logs@custom template. Make sure the backing index for the datastream uses “best_speed” as the index codec.
• [x] Update the logs@custom template to override the @timestamp field type using “date” (overriding the default “date_nanos”). Create a new datastream making use of the default logging templates and the logs@custom template. Make sure the backing index for the datastream uses “date_nanos” as the @timestamp field type.
• [x] Update the logs@custom template to override the index sort settings using a set of fields different from the default (host.name, @timestamp) and remove the @timestamp field. Make sure template update is successful even if a @timestamp field is missing. Create a new datastream making sure the index mapping includes a @timestamp field. The @timestampo field is automatically injected by Elasticsearch when using LogsDB as part of a datastream.
• [x] Update the logs@custom template to override the index sort settings using a set of fields different from the default (host.name, @timestamp) and replace the @timestamp field with a custom timestamp field. Make sure template update is successful even if a custom timestamp field is used. Create a new datastream making sure the index mapping includes a @timestamp field and the custom timestamp field and that index is sorted using the custom timestamp field. The @timestamp field is automatically injected by Elasticsearch when using LogsDB as part of a datastream.
• [x] Update the logs@custom template to override the ignore_malformed setting at index level and setting its value to “false” (default “true”). Make sure template update is successful. Create a new datastream making sure that the index settings includes the ignore_malformed setting and that the value is “false”, as overridden by logs@custom template.
• [ ] Update the logs@custom template to override the ignore_above setting at index level and setting its value to something different than 1024 (default 1024). Make sure template update is successful. Create a new datastream making sure that the index settings includes the ignore_above setting and that the value matches the expected value as overridden by logs@custom template.
• [x] Update the logs@custom template to override the ignore_dynamic_beyond_limit setting at index level and setting its value to “false” (default “true”). Make sure template update is successful. Create a new datastream making sure that the index settings includes the ignore_dynamic_beyond_limit setting and that the value is “false”, as overridden by logs@custom template.
• [x] Update the logs@custom template to disable doc values for every field type. Make sure template update fails while validating the template. LogsDB uses synthetic source as the only source mode and doc values are required. If doc values are disabled a stored field should be used under the hood to make sure documents are reconstructed including fields whose doc values are disabled.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[lkts]

ignore_above is not supported as index level setting so it is not covered.