PokeTCGData Workflows
Automated system to ingest daily Pokemon TCG pricing data from Pokemon TCG IO API.
The data is displayed in a live dashboard that can be found here. (Coming soon)
[!NOTE] The way this project is deployed/architected may not make sense due to budgeting and to better learn the applications being used.
Architecture
Tools
Database/Cloud Hosting
Setup
This does require a Azure SQL Database or Microsoft SQL Server instance to store data. There is usually a free tier for Azure SQL Database.
If you want to run this 100% locally, skip the steps related to terraform and AWS CLI. Also, set up a Microsoft SQL Server 2022 Linux container with Docker locally on your own as the database.
[!CAUTION] Although there are free tiers for Azure SQL and AWS EC2 instances, please be aware that costs may incur with these services.
I am not responsible for any costs that occur.
Prerequisites
- To run on local/development, install the following on your local machine:
- To deploy on AWS via Terraform, install the following on your local machine:
- Install Terraform CLI
- Create an AWS account
- Install and configure AWS CLI
How to start Airflow locally
There is a powershell script setup.sh that contains scripted functions to bring containers up, down, and restart them (locally).
👆 To bring up the environment, run the following command:
sh setup.sh start👇 To bring down the environment, run the following command:
sh setup.sh down🔄 To restart the environment, run the following command:
sh setup.sh restart