U.S. Weather Alerts Pipeline

Repo for the Le Wagon Data Engineering bootcamp final project

General project guidelines

• We use Github Flow for managing code contributions. For this reason, the main branch is protected from pushes. Create feature branches with the following naming convention: name-yyyymmdd-feature when for features you work on and create pull requests.
• Stand up each Tuesday and Thursday at 18:45 CET.

MVP Features

Screenshot from Slack

Architecture sketch

Local env Airflow and DBT conf :

Prerequisites :

Add the gcp keys to your VM (In the same location as the one your VM uses for the bootcamp) and name it as us_weather_alerts.json the path should be like the following

.gcp_keys/us_weather_alerts.json

Once the project is cloned in your local machine.

docker compose up --build

Useful Links

US National Weather Service API

• US National Weather Service API page
• National Weather Service API Python package
• US National Weather Service Glossary

Pandas-GBQ Package

• Writing Data to BigQuery
• Writing to an existing table
• Authenticating with a service account JSON