Data Engineering for Everybody
DE4E: Data Engineering for Everybody by Pseudo-Lab.
Loved the project? Please visit our Website
Welcome to our DE4E repository! We aim to give you a complete understanding of data engineering, from fundamentals to advanced concepts. Whether you're new or experienced, our repository empowers data lovers with the knowledge and skills for success in the data-driven era. Join us on this exciting journey as we unlock the full potential of data engineering together!
DE4E: Data Engineering for Everybody
click the above image will guide you to DE4E website
β‘οΈ move to the website: pseudo-lab.github.io/data-engineering-for-everybody
Acknowledgement π
DE4E: Data Engineering for Everybodyλ κ°μ§μ°κ΅¬μμ DSF νλ‘κ·Έλ¨μμ μμλμμ΅λλ€.
μμμ μμ κ°μ¬μ λ§μμ μ ν©λλ€.
κ°μ§μ°κ΅¬μλ DataCampμ νμμ λ°μ Donates νλ‘κ·Έλ¨μ μ§ννκ³ μμ΅λλ€. νλ‘κ·Έλ¨μ ν΅ν΄ ꡬμ§μ, λΆμμ μ·¨μ μ, λΉμ리 μ°κ΅¬ κ³Όνμ, νμλΆλ€κ» DataCampμμ μ 곡νλ λ€μν μ½μ€μ νΈλμ μ 곡ν©λλ€. λ³Έ νλ‘μ νΈλ DataCamp Donates νλ‘κ·Έλ¨ μ€ νλμΈ Data Science FellowshipμΌλ‘λΆν° μμλμμ΅λλ€.
DE4Eλ λ°μ΄ν° λΆμκ°, λ°μ΄ν° κ³Όνμ, λ°μ΄ν° μμ§λμ΄, λ¨Έμ λ¬λ μμ§λμ΄κ° ν¨κ» λͺ¨μ¬ λ°μ΄ν°μ, λ°μ΄ν°μ μν, λ°μ΄ν°λ₯Ό μν Data Engineering Repositoryλ₯Ό λ§λ€μ΄ λκ°κ³ μ ν©λλ€.
Contents
- Self-Check List
- Session 1. Introduction to Data Engineering
- Session 2. Data Sources and Data Collection
- Session 3. Data Transformation and Cleaning
- Session 4. Data Storage
- Session 5. Data Processing Frameworks
- Session 6. Data Processing Frameworks II
- [Session 7. Introduction to Apache Airflow]()
- Session 8. Cloud Computing and Data Engineering
- [Capstone Project(In Progress)]()
Schedule
| idx | Date | Subject | Presenter | Pre-Question | Tag |
|---|---|---|---|---|---|
| 0 | 2023-03-26 | Session 0. Orientation | μ΄μμ | Why should we learn Data Engineering? | #OT #Direction # Motivation |
| 1 | 2023-04-02 | Session 1. Introduction to Data Engineering | μ΄μμ | What is Data Engineering? | #Data Engineering #Discussion |
| 2 | 2023-04-09 | Session 2. Data Sources and Data Collection | μ΄λμ±, κΉμΈν | How can we collect data from variaty sources? | #Source Data #Data Collection #Data Type #Structured Data #Unstructured Data #Batch Data #Real-time Data |
| 3 | 2023-04-16 | Session 3. Data Transformation and Cleaning | μ΄μμ | How can we transform data more efficiently? | #Data Processing |
| 4 | 2023-04-30 | Session 4. Data Storage | μ‘μ€νΈ, μ ν¬μ | How can we store data more efficiently? | #Data Store #Database #Data Lake #Lakehouse #Object-Storage #NoSQL |
| 5 | 2023-05-07 | Session 5. Data Processing Frameworks | μ κ²½λ₯, μ΄νλ¦Ό | How data processing framework help us? | #Hadoop Eco-system #Parallel Computing |
| 6 | 2023-05-14 | Session 6. Data Processing Frameworks II | κΉμμ , μ΅νμΉ | Learn about various data processing framework | #Apache Spark #Apache Kafka #Apache Storm #Apache Flink |
| 7 | 2023-05-28 | Session 7. Introduction to Apache Airflow | κΉμ±ν, μ΄ν¬λ―Ό | How can we schedule, orchestrate data processing? | #Apache Airflow #Tutorial |
| 8 | 2023-06-04 | Session 8. Cloud Computing and Data Engineering | μ΄λ―Όν, μ΄μμ | What is Cloud Computing? and Why it is so important? | #Cloud Computing #Multi-Cloud #Data Engineering |
| 9 | 2023-06-18 | Capstone Project | μ΄νλ¦Ό | Let's dive into Data Engineering Capstone Proeject | #Capstone Project |
| 10 | 2023-07-09 - | Project Management | μ΄μμ , μ ν¬μ , μ κ²½λ₯, μ΄λμ±, κΉμμ | Build Together! | #Share #Motivation #Delighted to work together #Pseudo-Lab |
About us ππΌ
κ°μ§μ°κ΅¬μλ λ¨Έμ λ¬λ, λ°μ΄ν° μ¬μ΄μΈμ€, λ°μ΄ν° μμ§λμ΄λ§μ μ€μ¬μΌλ‘ λͺ¨μΈ λΉμ리λ¨μ²΄μ
λλ€. λꡬλ μνλ μ°κ΅¬λ₯Ό ν μ μλ μμμ μ΄ λλ, μ§μ§λ³΄λ€ λ μ§μ§ κ°μ μ°κ΅¬μλ₯Ό κΏκΎΈκ³ μμ΅λλ€. 곡μ (Share), λκΈ°λΆμ¬(Motivation), ν¨κ»νλ μ¦κ±°μ(Delighted to work together)λΌλ ν΅μ¬κ°μΉλ₯Ό μΆκ΅¬νλ©° μ½ 1800μ¬ λͺ
μ μ°κ΅¬μλΆλ€μ΄ μ€λλ ν¨κ» λ¨Έμ λ¬λ, λ°μ΄ν° μ¬μ΄μΈμ€, λ°μ΄ν° μμ§λμ΄λ§ λΆμΌμ μ ν μν₯λ ₯μ νμ¬νκ³ μμ΅λλ€. λ³΄λ€ μμΈν λ΄μ©μ μ¬κΈ°μ μ΄ν΄λ³΄μ€ μ μμ΅λλ€.
Contributors π
License π
This project is licensed under MIT license.