You can see the result in an end-to-end test here, that performs a write to the /api/v3/write API, and then performs a set of queries to verify that the data can be queried, and that the new line protocol is validated for correctness in the written series key.
Detailed Changes
Extend the Catalog to support tables with series key
After https://github.com/influxdata/influxdb/issues/25031, the TableDefinition in the catalog is just a wrapper around the Schema type from the core schema crate. Therefore, a lot of the following functionality stems from the changes made to that type to support the series key in influxdb3_core.
The catalog can support v1/v2 and `v3 tables simultaneously; the difference between them is:
v3 has a schema-level metadata entry that stores the series key members
v3 can contain Key columns, but not Tag columns, while v1/v2 is the inverse
A new order is enforced for columns in a schema. For v1/v2:
tags (in lexicographical order) -> fields -> time
For v3:
series keys (in user-defined order) -> fields -> time
Split the write path to enable validation and buffering of v1 and v3 writes
The code that parsed and validated incoming writes was previously written as a series of nested function calls. This PR refactored that code by:
Using a state machine WriteValidator to store the state that is re-used throughout the process. There are three steps to validation:
Load up the catalog, and create the database if it does not exist (fallible)
Validate the incoming line protocol, and update the relevant table schema (fallible)
Buffer the data (infallible)
Write separate methods on the WriteValidator implementation to do (i.) and (ii.) for v3
Add a new /api/v3/write API
This is the API that leverages the new write path, and can be used to perform writes using the new write protocol. It accepts the same parameters as the existing /api/v3/write_lp API.
hiltontj
commented
3 months ago
Closes #25033
Summary
This PR leverages a set of changes introduced experimentally into
influxdb3_coreto enable the v3 line protocol write API proposed in https://github.com/influxdata/influxdb/issues/24979.You can see the result in an end-to-end test here, that performs a write to the
/api/v3/writeAPI, and then performs a set of queries to verify that the data can be queried, and that the new line protocol is validated for correctness in the written series key.Detailed Changes
Extend the Catalog to support tables with series key
After https://github.com/influxdata/influxdb/issues/25031, the
TableDefinitionin the catalog is just a wrapper around theSchematype from the coreschemacrate. Therefore, a lot of the following functionality stems from the changes made to that type to support the series key ininfluxdb3_core.v1/v2and `v3 tables simultaneously; the difference between them is:v3has a schema-level metadata entry that stores the series key membersv3can containKeycolumns, but notTagcolumns, whilev1/v2is the inversev1/v2:For
v3:Split the write path to enable validation and buffering of
v1andv3writesThe code that parsed and validated incoming writes was previously written as a series of nested function calls. This PR refactored that code by:
WriteValidatorto store the state that is re-used throughout the process. There are three steps to validation:WriteValidatorimplementation to do (i.) and (ii.) forv3Add a new
/api/v3/writeAPIThis is the API that leverages the new write path, and can be used to perform writes using the new write protocol. It accepts the same parameters as the existing
/api/v3/write_lpAPI.