influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
28.19k stars 3.51k forks source link

Add an `/api/v3/write` API to use `v3` line protocol #25033

Closed hiltontj closed 2 weeks ago

hiltontj commented 1 month ago

A new write path needs to be set up to use the v3 line protocol, which is on an experimental branch in core: https://github.com/influxdata/influxdb3_core/pull/20

Ideally we should not have to rewrite the entire code path in the write_buffer that parses and validates incoming writes for the new API, as the framework should generally be the same. Currently, the differences between the existing write path and what will be the new write path are in the influxdb_line_protocol crate in core:

The v3 module introduced in influxdb_line_protocol has its own v3::parse_lines and v3::ParsedLine<'_> which will be used in the new write path.

hiltontj commented 4 weeks ago

We decided that instead of updating the existing /api/v3/write_lp API to use the new line protocol, that we would first add a new API to support the new v3 line protocol, hence the update to the issue title and description.

The code we have built around the existing API is substantial, i.e., in the influxdb3_client, as well as the load generator and our E2E tests, which use the client. Additionally, the /api/v3/write_lp as it is provides new functionality in the accept_partial parameter.

We can decide to stick to one API and do any necessary refactoring involved in #25034 and #25036, or proceed with two separate APIs.

pauldix commented 4 weeks ago

I think we'll want to proceed with a single API. Particularly given that we have another way already to write v1 LP into the DB. But in the interest of moving things forward I think it's best to keep this under a new experimental API endpoint until we've done a bit of testing and validation that it's what we want long term.