Epic: Sync time series data to a remote InfluxDB Instance

russorat commented 3 years ago

Target Customer / Background data / Context / Market Size

As a user, I have multiple edge deployments of InfluxDB OSS that are collecting data locally. I would like an easy way to analyze this data in aggregate.
- As a user, I have multiple edge deployments of InfluxDB OSS that are collecting data locally with limited or no connectivity to the cloud. I would like an easy way to analyze this data in aggregate once network connection is restored.

Goals

The goal of this epic is to provide a way for users to replicate data at the bucket level to one or more remote instances of InfluxDB

Acceptance Crit

as a user, i can configure an existing bucket to replicate all future points to one or more remote influxdb instances (add replication stream)
as a user, i can remove a replication stream from a bucket (delete replication stream)
as a user, i can list the replication streams for a bucket (list replication streams)
as a user, i can see information about how full the queues are when listing replication streams.
as a user, i can get a single replication stream by id for a bucket (get replication stream)
as a user, i can update a single replication stream for a bucket (update replication stream)
as a user, when i delete a bucket, i also delete any replication streams associated with that bucket
as a user, i can validate the connection information is valid before saving the replication streams
as a user, i can see status per replication stream indicating the connection is working
as a user, i can see log messages indicating that points are being dropped if the queue size is too large
as a user, i can see information about how full each queue is per replication stream per bucket (/metrics data)
as an analytics user, i can see information about replication streams in the phone home data
as an analytics user, the user agent should be set to something reasonable such as influxdb-oss/<version>

Release Plan

TBD

Early work goes behind a feature-flag so we can merge to master without fear of releasing?

Notes

the minimum connection details needed for a bucket replication stream are host, token, org (or orgID), bucket name (or bucket id), queue size (bytes), and skip-verify/insecure TLS.
- Dan opinion: would prefer to require IDs for both org and bucket; have heard rumors that Cloud APIs will move away from accepting names
- Russ: I'm ok with using only IDs for this for the first pass but there are a few issues. Today, the CLI configs file only stores the org name, not the ID, so we should explore adding orgid to that (and auto-populating it from the token if we can). That way the CLI could work with the user only specifying the local and remote bucket ids + queue size (although the user could send a write only token to the command for added security). For the UI, i imagine we could allow the user to browse the remote influxdb instance to select the bucket, in which case, we could use id as well. The only issue is that if the user drops and recreates the remote bucket, you could end up with failures.
Replication is configured at the bucket level via an API, which means a restart is not required
Configuration options for the size of the replication queue should be configurable per replication stream
Assumption: no support for injecting a query to filter/modify data before it's sent to the remote server
- If users want that behavior, they can have a task execute the query and write results to a 2nd local bucket, and replicate that 2nd bucket
Assumption: the auth tokens included in stream configuration are write-only (not included in list/GET metadata responses)
- We should only require write to the specific bucket. In the experiences we build, they should fail gracefully if the user doesn't have permissions to do things like list buckets.
Assumption(Sam): Only writes are streamed. Other data operations (e.g. deletes, backup restores) do not affect replication. This can cause data in the local bucket to be different than the remote.

Unanswered questions

What token permissions are needed to configuring replication?
Should we require an explicit toggle on buckets to enable/disable using them as a source for replication? Similar to how measurement schemas is enabled/disabled by the bucket-level "schema type" property
- Russ: i'm not sure why that would be necessary or desired. If users need the ability to "lock" a bucket so that it's only for replication, that can be added later IMO.
If data is sitting in a replication stream's queue when the stream is deleted, should the data still be flushed to the remote instance? Or deleted?
- Russ: Deletes should not be replicated to the remote bucket IMO. If a user needs to delete data from the remote bucket, they should run the same delete command on both instances. The reason a stream might be "full" and need to be deleted would be that the remote connection is misconfigured (or the bucket was recreated). In this case, i think deleting the stream should delete the data in the stream. Attempting a final flush makes sense here, but if the flush fails, the data should just be deleted. Updating the stream should be preferred when connection issues happen.
- Russ: This also means that users should validate that the streams are empty using the supplied APIs for viewing the stream size. I've added an acceptance criteria to return that information when listing streams.
Same question as ^^^ but for when the entire source bucket is deleted
- Russ: When a source bucket is deleted, the replication streams attached to that bucket should also be deleted. As stated above, deleting a replication stream should also delete the data in the stream.
Should it be possible to "pause" a replication stream without entirely deleting it?
- If a stream is paused when there is unflushed data in its queue, should the data still be sent? Or should it stay in the queue until the stream is un-paused / deleted?
- Russ: This feels like a follow on feature. Users could accomplish this today by marking the token inactive on the remote influxdb, which would cause the queue to fill up.
Should we confirm that points have been successfully written to the local DB before adding them to the replication queue?
- Russ: Since different influxdb instances use different storage engines, it's very possible a write could fail on one and succeed on the other. I need to think more about this, but i'm leaning towards the local instance api having the responsibility of validating the write as much as possible before accepting it. Local writes could still fail for things like disk space i guess, but in that case, it feels like allowing the data to continue to sync to the remote instance would be preferred.
Are there any cases when we would want points to be purged from a replication stream without successfully sync-ing to the remote instance? i.e. if we try sync-ing points and get back an error about conflicting schemas?
- Russ: I don't think being able to delete individual points from a replication stream makes sense. I do see a situation where we could continue to retry failed points forever though. I'm leaning towards the remote instance being responsible for anything where the payload was accepted but the write failed. See https://github.com/influxdata/docs-v2/issues/3003 . Different influxdb instances handle this differently though. I believe OSS will reject right away with a status code, while cloud will accept and reject later, storing the data in a log. Storing failed writes in a log might be something we should consider adding to OSS tbh. I'm leaning towards being very strict in what we keep in the queue and limiting it to purely connection, auth, and 500 errors. The reason being that in order to get into the queue in the first place, the write had to be accepted by the original API, so if it is rejected later, it would be due to conflicting types (oss would reject right away, cloud would accept then reject later) or something wrong with the remote server (disk space, etc).
- Russ: All this to say, I could envision a reason for wanting to "drop the queue" for a replication stream. This could be added later though, with the work around being delete the stream and recreate.
Apart from the baseline URL/bucket+org IDs, what other info (if any) might we eventually want to display in the UI/CLI view of replication streams? Ex: total request count, queue size, last successful request, last failure, etc.?
- Russ: I think queue size, last status code, and optional last message (in case of an error) makes sense.

Team

Anchor / Dev Champion - @danxmoran
Product Manager - @russorat
Design - TBD

lesam commented 2 years ago

Added: Assumption(Sam): Only writes are streamed. Other data operations (e.g. deletes, backup restores) do not affect replication. This can cause data in the local bucket to be different than the remote.

williamhbaker commented 2 years ago

Most of the work for this feature is now complete. It can be tried out using the latest nightly builds on influxd and the influx CLI for managing replications/remotes.

Warning: Running the latest nightly build of influxd will apply migrations to any existing bolt/sqlite db files it is pointed at specific to replications. If you later start an "older" version of influxd (for example, the latest 2.1.1 release), it will fail to startup since these migrations don't exist in anything we have released yet. You could recover from this using the influxd downgrade command to revert the metadata dbs to the 2.0 version, although this will delete the 2.1 specific migration data...like notebooks and annotations. It probably goes without saying, but I'd strongly recommend not running the nightly with data that you don't want to lose!

Nightly influxd server builds are available here, following the general pattern of these URLs for your desired arch (there is no nightly docker image)

Nightly influx CLI builds are available at similar paths:

Most notable work in progress still:

~If there is a remote write in progress, the server won't shutdown gracefully. Will be fixed by https://github.com/influxdata/influxdb/pull/22958~
~If you do a very large write to a local OSS instance that would be rejected by cloud due to cloud-specific size limits, the remote write will never succeed. This will be fixed by https://github.com/influxdata/influxdb/compare/replications-batch-large-writes, which will split up large local writes into smaller remote writes.~
~Most of the metrics/logging is complete, but there are still some improvements to be made, see https://github.com/influxdata/influxdb/issues/22953~
~When we are ready to release this feature, we will remove the feature flagging on the server and CLI side of course~
See open issues in the epic for other remaining work

williamhbaker commented 2 years ago

I've updated my previous post to reflect the fact that all of the in-progress work that was still to be completed is now complete.

Also, we have removed the server and CLI feature flags for remotes/replications, so those flags do not have to be turned on if you are working with the latest nightlies. You should be able to use the CLI normally and see the commands listed in the top-level help, and start the server without the replicationStreamBackend flag (you will actually get an error if you try to use this flag since it doesn't exist anymore).

tcurdt commented 2 years ago

Is there a rough ETA for this? This is exactly what I was searching for.

lesam commented 2 years ago

It is available now in the nightly release and will be in the next OSS release.

influxdata / influxdb