feat(flow): `flush_flow` function

discord9 commented 1 month ago

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

added a admin function flush_flow, which will signal flow worker to flush flow immediately and write newest results to sink table.

flush_flow admin_fn, also tweak admin_fn macro to support it
refactor admin_fn macro and use as many full path as possible
FlowServiceOperator to support flush flow in context
added back sqlness tests and modified them with flush_flow

For ease of reviewing, the major changes are listed in the following files, others are just mostly adding new args when creating QueryEngine:

src/common/function/src/flush_flow.rs for impl flush_flow function in sql
src/flow/src/adapter/flownode_impl.rs to impl flush flow in flow worker, note it will first sleep for 100 ms now before doing anything to prevent running flush before inputs are send to flow
src/common/function/src/handlers.rs and src/operator/src/flow.rs which impl FlowServiceHandler for FlowServiceOperator
sqlness tests that change SQLNESS SLEEP <TIME> to select flush_flow('test_numbers_basic')*0;

Checklist
[ ] I have written the necessary rustdoc comments.
[ ] I have added the necessary unit tests and integration tests.
[x] This PR requires documentation updates.

Summary by CodeRabbit

New Features
- Introduced new FlowServiceHandler and FlowServiceOperator for improved flow management.
- Added functions for flushing flow data and enhanced asynchronous programming capabilities.
- New SQL scripts for basic flow and DataFrame function testing.
Bug Fixes
- Improved error handling and logging across various components.
Documentation
- Updated documentation for new functions and parameters in the admin_fn macro to include support for FlowServiceHandlerRef.
Chores
- Streamlined import statements and removed unused dependencies to enhance code clarity.

coderabbitai[bot] commented 1 month ago

[!IMPORTANT]

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

This update enhances the project's capabilities by introducing new features and improving existing functionality. Key changes include the addition of flow service handling, modifications to asynchronous functions, and updates to dependency management. These alterations aim to streamline data processing and error handling within the system.

Changes

File(s)	Change Summary
Cargo.toml, src/common/function/Cargo.toml	Updated `greptime-proto` dependency version; added `futures.workspace` and `futures-util.workspace`.
src/cmd/src/cli/repl.rs	Modified `create_query_engine` function to accept an additional parameter.
src/common/function/src/flush_flow.rs	Introduced a new `flush_flow` asynchronous function for flow data management.
src/common/function/src/handlers.rs	Added `FlowServiceHandler` trait for flushing operations in flow services.
src/common/function/src/lib.rs	New module `flush_flow` added to the library.
src/common/function/src/state.rs	Enhanced `FunctionState` struct with `flow_service_handler`.
src/query/src/query_engine.rs	Updated `QueryEngineFactory` constructors to include a new optional parameter for flow services.
src/operator/src/flow.rs	Defined `FlowServiceOperator` struct implementing `FlowServiceHandler`.
tests/cases/standalone/common/flow/flow_basic.sql	Introduced SQL scripts for creating tables and flows to process numerical data.
tests/cases/standalone/common/flow/show_create_flow.sql	Renamed tables and flows to reflect new naming conventions.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant FlowService
    participant QueryEngine
    participant DataStore

    User->>FlowService: Initiate flush operation
    FlowService->>QueryEngine: Request flush
    QueryEngine->>DataStore: Execute data flush
    DataStore-->>QueryEngine: Acknowledge completion
    QueryEngine-->>FlowService: Flush completed
    FlowService-->>User: Respond with status

🐇 In the meadow, I hop with glee,
New flows and functions, oh such a spree!
With futures added, we dance and play,
Flushing data brightens the day!
Hooray for changes, let’s celebrate,
In code we trust, our dreams await! 🌼

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

discord9 commented 1 month ago

@coderabbitai review

coderabbitai[bot] commented 1 month ago

Actions performed

Review triggered. > Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 54.35540% with 131 lines in your changes missing coverage. Please review.

Project coverage is 84.70%. Comparing base (f787265) to head (ce18dc2). Report is 13 commits behind head on main.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #4416 +/- ## ========================================== - Coverage 85.03% 84.70% -0.34% ========================================== Files 1075 1076 +1 Lines 192544 192585 +41 ========================================== - Hits 163739 163132 -607 - Misses 28805 29453 +648 ```

GreptimeTeam / greptimedb