jschalk / jaar

Cross-cultural communication engine built on customizable economic systems. Produces personalized agendas, calendars, finance reports for users.
0 stars 1 forks source link

175 enable documentation process #386

Closed jschalk closed 1 week ago

jschalk commented 1 week ago

Summary by Sourcery

Refactor the codebase to implement a new staging workflow for data transformation processes, including renaming methods, classes, and test cases to reflect the changes.

Enhancements:

Tests:

sourcery-ai[bot] commented 1 week ago

Reviewer's Guide by Sourcery

This PR implements a significant refactoring of the data transformation pipeline, primarily focusing on renaming components and adding new functionality for handling nub staging. The changes maintain the existing data flow while making the naming more consistent and adding support for a new brick format (br00114).

ER diagram for new brick format br00114

erDiagram
    br00114AbstractTable {
        String face_id PK
        Integer event_id PK
        String acct_id
        String fiscal_id
        String inx_label
        String otx_label
        String jaar_type
        String owner_id
    }

    br00114HoldTable {
        String face_id PK
        Integer event_id PK
        String acct_id
        String fiscal_id
        String inx_label
        String otx_label
        String jaar_type
        String owner_id
    }

    br00114StageTable {
        String face_id PK
        Integer event_id PK
        String acct_id
        String fiscal_id
        String inx_label
        String otx_label
        String jaar_type
        String owner_id
        String src_type
        String src_path
        String src_sheet
    }

    br00114AbstractTable ||--|| br00114HoldTable : "extends"
    br00114AbstractTable ||--|| br00114StageTable : "extends"

Updated class diagram for data transformation classes

classDiagram
    class JungleToZooTransformer {
        +transform()
        +_group_jungle_data()
        +_read_and_tag_dataframe(ref)
        +_save_to_zoo_staging(brick_number, dfs)
    }

    class ZooStagingToZooAggTransformer {
        +transform()
        +_group_by_brick_columns(zoo_staging_df, brick_number)
        +_save_zoo_agg(brick_path, zoo_staging_df)
    }

    class ZooAggToZooEventsTransformer {
        +transform()
        +get_unique_events(zoo_agg_df)
        +_save_zoo_events(brick_path, events_df)
    }

    class ZooEventsToEventsLogTransformer {
        +transform()
        +get_event_log_df(otx_events_df, x_dir, x_file_name)
        +_save_events_log_file(events_df)
    }

    class ZooAggToOtx2InxStagingTransformer {
        +transform()
        +insert_legitmate_zoo_agg_otx2inx_atts(otx2inx_df, brick_number, zoo_brick_path, otx2inx_columns)
    }

    class ZooAggToNubStagingTransformer {
        +transform()
        +insert_legitmate_zoo_agg_nub_atts(nub_df, brick_number, zoo_brick_path, nub_columns)
    }

File-Level Changes

Change Details Files
Renamed data transformation pipeline components to use more consistent terminology
  • Renamed 'zoo' to 'zoo_staging' for initial data stage
  • Renamed 'otx' to 'zoo_agg' for aggregated data
  • Renamed 'otx_events' to 'zoo_events' for event data
  • Updated class names to reflect new terminology (e.g., ZooToOtxTransformer -> ZooStagingToZooAggTransformer)
src/f10_world/world.py
src/f10_world/test/test_world_jungle_to_zoo.py
Added new brick format br00114 for account nub label handling
  • Created new brick format JSON configuration file
  • Added br00114 table definitions to brick models
  • Updated brick category references to include br00114
  • Added br00114 to brick format headers
src/f09_brick/brick_formats/brick_format_00114_acct_nub_label_v0_0_0.json
src/f09_brick/brick_models.py
src/f09_brick/brick_config.py
Implemented new nub staging transformation functionality
  • Added ZooAggToNubStagingTransformer class
  • Implemented zoo_agg_to_nub_staging method in WorldUnit
  • Added tests for nub staging functionality
src/f10_world/world.py
src/f10_world/test/test_world_jungle_to_zoo.py

Tips and commands #### Interacting with Sourcery - **Trigger a new review:** Comment `@sourcery-ai review` on the pull request. - **Continue discussions:** Reply directly to Sourcery's review comments. - **Generate a GitHub issue from a review comment:** Ask Sourcery to create an issue from a review comment by replying to it. - **Generate a pull request title:** Write `@sourcery-ai` anywhere in the pull request title to generate a title at any time. - **Generate a pull request summary:** Write `@sourcery-ai summary` anywhere in the pull request body to generate a PR summary at any time. You can also use this command to specify where the summary should be inserted. #### Customizing Your Experience Access your [dashboard](https://app.sourcery.ai) to: - Enable or disable review features such as the Sourcery-generated pull request summary, the reviewer's guide, and others. - Change the review language. - Add, remove or edit custom review instructions. - Adjust other review settings. #### Getting Help - [Contact our support team](mailto:support@sourcery.ai) for questions or feedback. - Visit our [documentation](https://docs.sourcery.ai) for detailed guides and information. - Keep in touch with the Sourcery team by following us on [X/Twitter](https://x.com/SourceryAI), [LinkedIn](https://www.linkedin.com/company/sourcery-ai/) or [GitHub](https://github.com/sourcery-ai).