TSMC Cloud Native class final project: Digital Business
A solution for collecting Internet volume for some key words with CI/CD workflow automation.
Architecture
Components
Crawler Scheduler
- Cronjob to generate URL
- Send URLs to RabbitMQ
RabbitMQ
- Queue URLs
Crawler
- Consume URLs from RabbitMQ
- Parse URL data
- Generate result and store into InfluxDB
InfluxDB
- Keep data from Crawler
Grafana
- Read data from InfluxDB
- Show the result figure
CI/CD Workflow Design
Dev branch for development, with a CI workflow for test/build (push) images to dockerhub; after some feature or bug released/fixed, contributor can raise a pull request merge dev branch to master.
The pull request for master branch will need at least one reviewer approval comment to accept the request.
After merge the request to master, ArgoCD will sync the application status to cluster.
Tools
Github Action use for CI workflow, run pytest and build (push) image to dockerhub.
ArgoCD use for install helm charts on GCP cluster.
Application Present
(Grafana dashboard)
