Closed jschalk closed 4 days ago
This pull request implements a significant refactoring of the world module's ETL process, focusing on improving the bridge functionality and file organization. The changes include renaming methods for better clarity, introducing new brick formats, and restructuring test files.
erDiagram
br00115AbstractTable {
String face_id PK
Integer event_id PK
String acct_id
String fiscal_id
String inx_group_id
String otx_group_id
String owner_id
}
br00116AbstractTable {
String face_id PK
Integer event_id PK
String acct_id
String fiscal_id
String inx_node
String otx_node
String owner_id
}
br00117AbstractTable {
String face_id PK
Integer event_id PK
String acct_id
String fiscal_id
String inx_road
String otx_road
String owner_id
}
br00115AbstractTable ||--o{ br00115HoldTable : ""
br00116AbstractTable ||--o{ br00116HoldTable : ""
br00117AbstractTable ||--o{ br00117HoldTable : ""
br00115AbstractTable ||--o{ br00115StageTable : ""
br00116AbstractTable ||--o{ br00116StageTable : ""
br00117AbstractTable ||--o{ br00117StageTable : ""
classDiagram
class ZooAggToStagingTransformer {
+zoo_dir: str
+pidgin_category: str
+legitmate_events: set[TimeLinePoint]
+jaar_type: str
+transform()
+insert_staging_rows(stage_df: DataFrame, brick_number: str, zoo_brick_path: str, df_columns: list[str])
+get_sheet_staging_name() str
+get_otx_obj(x_row) str
+get_inx_obj(x_row, missing_col: set[str]) str
}
Change | Details | Files |
---|---|---|
Refactored the ETL process by splitting and renaming methods |
|
src/f10_world/world.py |
Added new brick format definitions and models |
|
src/f09_brick/brick_models.py src/f09_brick/brick_config.py src/f09_brick/brick_formats/brick_format_00115_group_bridge1_v0_0_0.json src/f09_brick/brick_formats/brick_format_00116_node_bridge1_v0_0_0.json src/f09_brick/brick_formats/brick_format_00117_road_bridge1_v0_0_0.json |
Reorganized and renamed test files for better organization |
|
src/f10_world/test/test_world_etl0_jungle_to_zoo.py src/f10_world/test/test_world_etl1_zoo_otx_appt.py src/f10_world/test/test_world_etl1_zoo_otx_group.py src/f10_world/test/test_world_etl1_zoo_otx_node.py src/f10_world/test/test_world_etl1_zoo_otx_nub.py src/f10_world/test/test_world_etl1_zoo_otx_road.py |
Improved naming consistency in bridge and pidgin modules |
|
src/f08_pidgin/bridge.py src/f08_pidgin/pidgin.py src/f08_pidgin/test_pidgin/test_pidginunit_crud_node.py |
Summary by Sourcery
Enable the documentation process by introducing new brick formats and corresponding table classes for group, node, and road bridges. Refactor the staging transformer to support multiple pidgin categories and update tests to cover new scenarios.
New Features:
Enhancements:
Tests: