bacalhau-project / bacalhau

Compute over Data framework for public, transparent, and optionally verifiable computation
https://docs.bacalhau.org
Apache License 2.0
706 stars 89 forks source link

decouple envelope from ncl #4695

Closed wdbaruni closed 2 weeks ago

wdbaruni commented 2 weeks ago

In preparation to utilize JetStream in NCL, it was needed to decouple non-message passing related logic from NCL, mainly message serialization and encoding logic, which is now moved to its own envelope library, which can be used by job store and other components as well to encode entries

Summary by CodeRabbit

coderabbitai[bot] commented 2 weeks ago

Walkthrough

The changes in this pull request involve a comprehensive transition from the ncl package to the envelope package across multiple files. This includes updating method signatures, changing message types, and modifying how messages are created and handled within various components such as message handlers, dispatchers, and test suites. The updates affect both the logic of message processing and the associated metadata handling, ensuring a consistent use of the envelope package throughout the codebase.

Changes

File Path Change Summary
pkg/compute/message_handler.go Updated MessageHandler methods to use envelope.Message instead of ncl.Message. Changed metadata key retrieval to envelope.KeyMessageType. Modified error handling for unexpected payload types.
pkg/compute/watchers/ncl_dispatcher.go Altered NCLDispatcher to utilize envelope.Message. Updated message creation in HandleEvent to use envelope.NewMessage. Added import for envelope package.
pkg/compute/watchers/ncl_dispatcher_test.go Changed test suite to handle *envelope.Message instead of *ncl.Message. Updated serializer and registry to use envelope methods.
pkg/lib/envelope/README.md Added documentation for the Envelope Library, detailing its features and usage.
pkg/lib/envelope/errors.go Changed package name from ncl to envelope. Added new error constants and retained existing error types.
pkg/lib/envelope/interface.go Introduced MessageSerializer and PayloadSerializer interfaces for handling message serialization.
pkg/lib/envelope/json.go Added JSON serialization types and methods for handling messages in JSON format.
pkg/lib/envelope/json_test.go Introduced unit tests for JSON serialization functionalities.
pkg/lib/envelope/message.go Added EncodedMessage and Message types for structured message handling.
pkg/lib/envelope/message_test.go Updated package name to envelope and retained test suite structure for message functionalities.
pkg/lib/envelope/metadata.go Defined Metadata type and methods for managing message metadata.
pkg/lib/envelope/proto.go Introduced ProtoMessageSerializer for Protocol Buffers serialization.
pkg/lib/envelope/proto_test.go Added test suite for ProtoMessageSerializer.
pkg/lib/envelope/registry.go Refactored MessageSerDeRegistry to Registry with updated methods for message handling.
pkg/lib/envelope/registry_test.go Updated test suite to reflect changes in registry implementation.
pkg/lib/envelope/serializer.go Renamed and refactored serialization logic to use envelope types.
pkg/lib/envelope/serializer_test.go Updated test suite for the new serializer implementation.
pkg/lib/envelope/utils_test.go Added utility types for testing within the envelope package.
pkg/lib/envelope/version.go Changed package name from ncl to envelope, retaining constants and methods.
pkg/lib/envelope/version_test.go Updated package name and retained test suite for schema version constants.
pkg/lib/ncl/README.md Modified documentation for clarity and conciseness.
pkg/lib/ncl/checkpointer.go Updated Checkpoint method to accept *envelope.Message.
pkg/lib/ncl/constants.go Added new constants for metadata keys.
pkg/lib/ncl/filter.go Updated ShouldFilter method to accept *envelope.Metadata.
pkg/lib/ncl/message.go Deleted file containing Message and Metadata implementations.
pkg/lib/ncl/message_serde.go Deleted file for JSON serializer/deserializer for messages.
pkg/lib/ncl/message_serde_test.go Deleted test suite for message serializers.
pkg/lib/ncl/publisher.go Updated publisher struct to use envelope types for message serialization.
pkg/lib/ncl/publisher_test.go Modified test suite to align with the new message handling logic.
pkg/lib/ncl/raw_message_serde.go Deleted file for raw message serialization.
pkg/lib/ncl/raw_message_serde_test.go Deleted test suite for raw message serialization.
pkg/lib/ncl/subscriber.go Updated subscriber struct to use envelope types for message deserialization.
pkg/lib/ncl/subscriber_test.go Modified test suite to reflect changes in message handling.
pkg/lib/ncl/types.go Updated method signatures to use envelope.Message and envelope.Metadata.
pkg/lib/ncl/utils_test.go Updated test handler to process *envelope.Message.
pkg/node/compute.go Updated NewComputeNode to accept *envelope.Registry.
pkg/node/heartbeat/client.go Changed message creation in SendHeartbeat to use envelope.NewMessage.
pkg/node/heartbeat/heartbeat_test.go Updated test suite to use envelope.Registry.
pkg/node/heartbeat/server.go Modified message handling methods to use envelope.Message.
pkg/node/ncl.go Updated CreateMessageSerDeRegistry to return *envelope.Registry.
pkg/node/requester.go Updated NewRequesterNode to accept *envelope.Registry.
pkg/orchestrator/message_handler.go Updated MessageHandler methods to use envelope.Message.
pkg/orchestrator/message_handler_test.go Modified tests to use envelope package for message handling.
pkg/orchestrator/watchers/ncl_dispatcher.go Updated dispatcher methods to create messages with envelope.NewMessage.
pkg/orchestrator/watchers/ncl_dispatcher_test.go Changed test suite to handle *envelope.Message.

Poem

In the meadow where messages hop,
The envelope now takes the top.
With each new type, the code does dance,
A leap to clarity, a joyful chance!
From ncl to envelope, we cheer,
For messages bright, let’s all draw near! 🐇✨


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share - [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)
🪧 Tips ### Chat There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai): - Review comments: Directly reply to a review comment made by CodeRabbit. Example: - `I pushed a fix in commit , please review it.` - `Generate unit testing code for this file.` - `Open a follow-up GitHub issue for this discussion.` - Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples: - `@coderabbitai generate unit testing code for this file.` - `@coderabbitai modularize this function.` - PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples: - `@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.` - `@coderabbitai read src/utils.ts and generate unit testing code.` - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.` - `@coderabbitai help me debug CodeRabbit configuration file.` Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. ### CodeRabbit Commands (Invoked using PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai full review` to do a full review from scratch and review all the files again. - `@coderabbitai summary` to regenerate the summary of the PR. - `@coderabbitai resolve` resolve all the CodeRabbit review comments. - `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository. - `@coderabbitai help` to get help. ### Other keywords and placeholders - Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. - Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description. - Add `@coderabbitai` anywhere in the PR title to generate the title automatically. ### CodeRabbit Configuration File (`.coderabbit.yaml`) - You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository. - Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information. - If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json` ### Documentation and Community - Visit our [Documentation](https://coderabbit.ai/docs) for detailed information on how to use CodeRabbit. - Join our [Discord Community](http://discord.gg/coderabbit) to get help, request features, and share feedback. - Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.