y-scope / clp

Compressed Log Processor (CLP) is a free log management tool capable of compressing logs and searching the compressed logs without decompression.
https://yscope.com
Apache License 2.0
871 stars 70 forks source link

ffi: Add support for serializing/deserializing auto-generated and user-generated schema tree node IDs. #557

Closed LinZhihao-723 closed 1 month ago

LinZhihao-723 commented 1 month ago

Description

As explained in #556, we want to add support for auto-generated key-value pairs in our current kv pair IR format. This PR implements one's complement encoding to serialize/deserialize encoded schema tree node IDs, and use these new methods to replace the old methods used in the serializer/deserializer implementations.

The serialization/deserialization methods in this PR use templates to reduce code duplication but without introducing runtime overhead. The template is mainly used for:

  1. To differentiate and serialize auto-generated and user-generated node IDs accordingly (determined in compile time)
  2. To adapt different encoding length indicator tags used for parent node ID serializations and key node ID serialization

Notice that the current implementation only uses user-generated node IDs' code path as the auto-generated node IDs are not yet supported. However, this PR changes the IR formats, meaning that the IR format serialized by the old serializer may not be correctly processed by the latest deserializer. As a result, this PR elevates the IR stream version number to 0.1.0-beta.1.

Validation performed

Summary by CodeRabbit

coderabbitai[bot] commented 1 month ago

Walkthrough

The pull request introduces several modifications to the clp::ffi::ir_stream namespace, enhancing serialization and deserialization functionalities. Key updates include the addition of a templated function in the Serializer class for handling schema tree node IDs, improvements to deserialization methods with updated return types for better error handling, and the introduction of new utility functions. Additionally, protocol constants have been renamed for clarity, and a new template test case has been added to validate the serialization logic.

Changes

File Path Change Summary
components/core/src/clp/ffi/ir_stream/Serializer.cpp Refactored serialization methods, introduced encode_and_serialize_schema_tree_node_id, updated serialize_schema_tree_node and serialize_key methods.
components/core/src/clp/ffi/ir_stream/Serializer.hpp Updated method signatures and documentation for clarity on return values.
components/core/src/clp/ffi/ir_stream/ir_unit_deserialization_methods.cpp Modified return types for deserialization functions, added is_encoded_key_id_tag and is_log_event_ir_unit_tag, refactored deserialize_schema.
components/core/src/clp/ffi/ir_stream/ir_unit_deserialization_methods.hpp Updated method signatures to enhance error handling.
components/core/src/clp/ffi/ir_stream/protocol_constants.hpp Updated constants for versioning and encoding identifiers, including renaming for clarity.
components/core/src/clp/ffi/ir_stream/utils.cpp Enhanced error handling functions, added new template functions for serialization and deserialization.
components/core/src/clp/ffi/ir_stream/utils.hpp Added new template functions to improve handling of schema tree node IDs.
components/core/tests/test-ir_encoding_methods.cpp Added new template test case ffi_ir_stream_serialize_schema_tree_node_id for validating serialization and deserialization of schema tree node IDs.

Possibly related PRs

Suggested reviewers


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share - [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)
🪧 Tips ### Chat There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai): - Review comments: Directly reply to a review comment made by CodeRabbit. Example: - `I pushed a fix in commit , please review it.` - `Generate unit testing code for this file.` - `Open a follow-up GitHub issue for this discussion.` - Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples: - `@coderabbitai generate unit testing code for this file.` - `@coderabbitai modularize this function.` - PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples: - `@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.` - `@coderabbitai read src/utils.ts and generate unit testing code.` - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.` - `@coderabbitai help me debug CodeRabbit configuration file.` Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. ### CodeRabbit Commands (Invoked using PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai full review` to do a full review from scratch and review all the files again. - `@coderabbitai summary` to regenerate the summary of the PR. - `@coderabbitai resolve` resolve all the CodeRabbit review comments. - `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository. - `@coderabbitai help` to get help. ### Other keywords and placeholders - Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. - Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description. - Add `@coderabbitai` anywhere in the PR title to generate the title automatically. ### CodeRabbit Configuration File (`.coderabbit.yaml`) - You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository. - Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information. - If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json` ### Documentation and Community - Visit our [Documentation](https://coderabbit.ai/docs) for detailed information on how to use CodeRabbit. - Join our [Discord Community](http://discord.gg/coderabbit) to get help, request features, and share feedback. - Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.