search

iluwatar / java-design-patterns

Design patterns implemented in Java
https://java-design-patterns.com
Other
88.2k stars 26.17k forks source link

Extract-Transform-Load (ETL) architectural pattern #328

Open iluwatar opened 8 years ago

iluwatar commented 8 years ago

Description: The Extract-Transform-Load (ETL) design pattern is crucial for data integration and data warehousing processes. It involves extracting data from various sources, transforming it to fit operational needs, and loading it into a target database or data warehouse. This pattern ensures that data is accurately consolidated and structured for analysis and reporting purposes.

Main Elements of the ETL Design Pattern:

  1. Extract: Retrieve data from different sources, which can include databases, APIs, flat files, etc.
  2. Transform: Cleanse, format, and structure the extracted data to meet the required specifications. This might involve data validation, normalization, aggregation, and other data transformation techniques.
  3. Load: Insert the transformed data into a target system, such as a database or data warehouse, ensuring it is ready for use in analysis and reporting.

References:

  1. ETL Design Pattern - Wikipedia
  2. What is ETL (Extract Transform Load)?

Acceptance Criteria:

  1. Implement a basic ETL framework that includes modules for data extraction, transformation, and loading.
  2. Provide examples demonstrating how to use the ETL framework with different data sources and targets.
  3. Ensure the implementation follows the project’s coding standards and includes comprehensive documentation and unit tests.
natashasrivastava commented 1 year ago

https://upload.wikimedia.org/wikipedia/commons/d/d8/ETL_Architecture_Pattern.jpg @iluwatar is not found. Also, I would like to work on it.

iluwatar commented 1 year ago

Assigned to you @natashasrivastava. Sorry about the link that has stopped working, but I'm sure you can find plenty more references by searching for ETL pattern

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

takikuba commented 5 months ago

I will take care of it.

stale[bot] commented 2 months ago

This issue has been automatically marked as stale because it has not had recent activity. The issue will be unassigned if no further activity occurs. Thank you for your contributions.

iluwatar commented 4 weeks ago

Updated task description