This repository features self-contained, hands-on-labs with detailed and step-by-step instructions, associated collateral (data, code, configuration, terraform etc) that demystify products, features and integration points to architect and build modern datalakes on GCP.
| # | Use Case | Lab summary | Author |
|---|---|---|---|
| 1. | Admin Use Case | GCP Infrastructure Provisioning via Terraform and Cloud Shell | |
| 2. | Data Engineer Use Case | Running Spark workloads on Dataproc-GCE and Dataproc-Serverless; Read data from GCS,uses Dataproc Metastore service for metadata, processes data and persists to BigQuery | |
| 3. | Data Scientist Use Case | Spark based MLOps at scale powered by Dataproc Serverless Spark for scalable ML training and inferencing, Vertex AI Pipelines for orchestration of ML tasks, with Vertex AI Workbench notebooks for experimentation/IDE | |
| 4. | Big Lake Fine Grained Access Control Use Case | Implementing fine-grained access control on data lakes, made possible by BigLake with a minimum viable example of Icecream sales forecasting on a Spark notebook hosted on a personal auth Cloud Dataproc GCE cluster. | |
| 5. | Delta Lake Table Format Lab | This lab aims to demystify Delta Lake with Apache Spark on Cloud Dataproc, with a minimum viable sample of the core features of the table format on a Data Lake on Cloud Storage. | |
| 6. | Hive to BigQuery with BigLake migration | This workshop contains a step by step demo that shows to migrate a HIVE workload from a on-prem Hadoop deployment based on legacy Cloudera 5.7 distribution to a Google Cloud modern DataLake. | |
| 7. | From Data Analysis with pandas to Data Engineering with SPARK | This workshop contains a step by step demo that shows a typical Data Analytics user journey where data analysts explore and analyze data using python (pandas) and then the data engineer teams formalizes the code using scalable frameworks like SPARK. | |
| 8. | SPARK ETL with RAPIDS | The repository shows a realistic ETL workflow with dataproc using RAPIDs |
See the contributing instructions to get started contributing.
All solutions within this repository are provided under the Apache 2.0 license. Please see the LICENSE file for more detailed terms and conditions.
This repository and its contents are not an official Google Product.
Share you feedback, ideas, by logging issues.
| # | Release Summary | Date | Contributor |
|---|---|---|---|
| 1. | Initial release | 12/02/2022 | Various |
| 2. | |||
| 3. |