[Feature] Export Data Based on Backup Snapshots

[hanahmily]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Description

Description

This feature relies on the Backup and Restore Data functionality outlined in #12463. The backup functionality creates a consistent snapshot of the dataset. Building on that, the Export Manager will enable exporting data based on these snapshots, allowing users to extract and utilize specific sets of data efficiently.

Requirements

1. Integration with Backup Snapshots:

   • Leverage the snapshot created by the Backup and Restore feature as the data source for export.
   • Ensure the export process clean up the snapshot is created for the exportation.

2. Export Configuration:

   • Time Range Export: Allow users to define a specific time range from the snapshot for export.
   • Query-Based Export: Allow users to filter and export data using specific query criteria (e.g., tags, metrics, identifiers).

3. Export Formats:

   • Support commonly used data formats like JSON and CSV.

Use case

No response

Related issues

No response

Are you willing to submit a pull request to implement this on your own?

• [ ] Yes I am willing to submit a pull request on my own!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[wu-sheng]

Is Export Manager going to be a separate role node?

[hanahmily]

Is Export Manager going to be a separate role node?

It is a command-line tool provided by bydbctl, which can be scheduled to run automatically using cron on a daily basis or at specified intervals.

[wu-sheng]

So, the tool is going to grap snapshots from mutiple data nodes(through liaison)? Where the filter happens? Are they processed on data nodes?

[wu-sheng]

About the feature, the exportation should support tags/fields selection if possible. This could reduce the volume costs of the files.

[hanahmily]

I want to create a design to explain the details. Perhaps we can set up exporting to the backup process.