3 implement rules engine to process data comparison of dataframes

[mlaguren]

Description

Incorporated Durable Rules as a rules engine to implement the following rules for comparison of Dataframes:

1. Compare # of rows
2. Compare cell values between the two DataFrames
3. Check if target DataFrames does not have an empty cell

Fixes # (issue)

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

How Has This Been Tested?

• Updated data validation tests that demonstrates the comparison rules
• Increase unit test coverage of modules/functions

Checklist:

• [x] My code follows the style guidelines of this project
• [x] I have performed a self-review of my code
• [x] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [x] My changes generate no new warnings
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] New and existing unit tests pass locally with my changes
• [ ] Any dependent changes have been merged and published in downstream modules

Summary by CodeRabbit

• New Features

  • Introduced Durable Rules for data comparison.
  • Added predefined rules for data comparison (equal_row_count, rows_match, not_null).
  • Expanded roadmap with custom rule creation, preprocessors for data manipulation, additional connectors, and content pages.

• Documentation

  • Enhanced class and method documentation across various connectors and services.

• Tests

  • Added extensive test cases for comparison rules, connectors, and data processing functionalities.
  • Introduced pytest fixtures and mock implementations for testing purposes.

[mlaguren]

@coderabbitai review command

[coderabbitai[bot]]

@mlaguren, I will review the changes made in the pull request regarding the implementation of the rules engine for data comparison. This review will cover the code style, functionality, and unit tests associated with the new features.

Actions performed

Review triggered. > Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai[bot]]

[!IMPORTANT]

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

Recent updates introduce several new features and enhancements, primarily focusing on improving the data validation and comparison framework. Notable additions include the integration of Durable Rules, expanded roadmap goals for better data manipulation and connectivity, detailed docstrings for various classes, and new test cases to ensure robust functionality across multiple components.

Changes

Files	Change Summaries
.gitignore	Added .coverage and allure-results to be ignored.
README.md	Added details on Durable Rules, predefined rules for data comparison, and expanded the project's roadmap.
data_validation_tests/...	Modified comparison rules for CSV to CSV and CSV to PostgreSQL examples to use new rules equal_row_count, rows_match, and not_null.
libs/comparison_rules.py	Introduced a new ruleset with functions for comparing dataframes using Durable Rules for checking equal row counts, matching rows, and detecting null values.
libs/connector_factory.py	Updated with detailed docstrings and added validation for connector types ('csv', 'postgres').
libs/csv_connector.py	Added detailed docstrings for the CSVConnector class, explaining attributes and methods.
libs/data_connector.py	Added detailed method documentation for the DataConnector class, providing descriptions for connect, fetch_data, and close methods.
libs/data_service.py	Enhanced the DataService class with detailed docstrings for its description, attributes, and the process_data method.
libs/postgres_connector.py	Added comprehensive docstrings and improved methods for the PostgresConnector class, including connection handling and data fetching.
main.py	Updated functionality for comparing dataframes, leveraging the new rules from libs.comparison_rules and durable.lang.
requirements.txt	Added new dependencies and upgraded existing ones to newer versions.
tests/conftest.py	Introduced a pytest fixture rule_engine_state() to create a fresh state for each test.
tests/test_check_keys.py	Enhanced test descriptions for check_keys_in_list function behavior.
tests/test_comparison_rules.py	Added test cases for new comparison rules such as rows_match, not_null, and null.
tests/test_connector_factory.py	Added test cases for validating ConnectorFactory behavior for different connector types and handling invalid types.
tests/test_csv_connector.py	Introduced test cases for CSVConnector methods ensuring proper functionality and interactions.
tests/test_data_connector.py	Added test cases to validate instantiation scenarios for the DataConnector class.
tests/test_data_service.py	Included a test case for the process_data method in DataService using a mock DataConnector.
tests/test_postgres_connector.py	Introduced a pytest fixture for mocking PostgresConnector and included a test case for the close method.
tests/test_process_data.py	Added test cases for process_data function using different data connectors (CSV, PostgreSQL) with mock objects.

🌟 In the land of code, our framework grows, Durable Rules now guide the flows. Data compares with rigour and might, Connectors are clear, gleaming bright. Testing's robust, coverage grand, In this digital realm, everything's planned. 🐇🚀

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

Tips

### Chat There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai): - Review comments: Directly reply to a review comment made by CodeRabbit. Example: - `I pushed a fix in commit .` - `Generate unit testing code for this file.` - `Open a follow-up GitHub issue for this discussion.` - Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples: - `@coderabbitai generate unit testing code for this file.` - `@coderabbitai modularize this function.` - PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples: - `@coderabbitai generate interesting stats about this repository and render them as a table.` - `@coderabbitai show all the console.log statements in this repository.` - `@coderabbitai read src/utils.ts and generate unit testing code.` - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.` - `@coderabbitai help me debug CodeRabbit configuration file.` Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. ### CodeRabbit Commands (invoked as PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai full review` to do a full review from scratch and review all the files again. - `@coderabbitai summary` to regenerate the summary of the PR. - `@coderabbitai resolve` resolve all the CodeRabbit review comments. - `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository. - `@coderabbitai help` to get help. Additionally, you can add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. ### CodeRabbit Configration File (`.coderabbit.yaml`) - You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository. - Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information. - If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json` ### Documentation and Community - Visit our [Documentation](https://coderabbit.ai/docs) for detailed information on how to use CodeRabbit. - Join our [Discord Community](https://discord.com/invite/GsXnASn26c) to get help, request features, and share feedback. - Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.