search

josephmachado / data_engineering_best_practices

Sample project to demonstrate data engineering best practices
https://www.startdataengineering.com/post/de_best_practices/
154 stars 24 forks source link

Build cloud infrastructure #4

Open josephmachado opened 1 year ago

josephmachado commented 1 year ago

Description

Currently the project is only meant to be run locally, implement a solution to deploy this to the cloud.

Areas to cover

  1. Platform to use
  2. CI/CD
  3. Env variable management (w Secret managements)
  4. Cloud storage & processing.
  5. Scheduling & Orchestration.
  6. Logging, metadata & debugging

Choice of platform

Please leave a comment with your choices for areas 1 - 5 above.

andreale28 commented 1 year ago

Hi Joseph

Thanks for sharing your project.

In case of next area to cover, I would love to see the expansion of this project to orchestration and/or platform. Otherwise, how about some talk on open table format?

abdelhaqs commented 12 months ago

All the topics mentioned above are interesting to see.

flaviassantos commented 10 months ago

Great innitiative @josephmachado! I would love to see more about points 2,3&5. :)

cjj1120 commented 9 months ago

I would like to further explore 4&5 those two provide the best value for me as a data engineer. I would be glad to support if you can help with those two! Great job and thanks for the website, appreciate it.

kottargiveer commented 9 months ago

Great Project. Thank you.

However, It would be great, if you could teach us how to create such projects from scratch.

How to create those containers so that we can ship projects from dev to QA to PROD ?

ex : The underlying script for these commands make up # Spin up containers make ddl # Create tables & views make ci # Run checks & tests make etl # Run etl make spark-sh # Spark shell to check created tables

cjj1120 commented 6 months ago

@kottargiveer The underlying commands are all under the Makefile, if that's what you're asking.

cjj1120 commented 6 months ago

My top 3 choices: 4 -> 5 -> 2

GEJ1 commented 5 months ago

Great project, I would be interested in all the topics mentioned above. Thank you very much!