Create a Argo Workflow for MLOps

ClimateMisinformation / Scrapers

Web scrapers

5 stars 1 forks source link

Create a Argo Workflow for MLOps #25

Open ebrucucen opened 3 years ago

ebrucucen commented 3 years ago

We want a repeatable and reliable process for data ingestion. The options we had were:

Kubeflow (YAML based)
Cloud Composer ( use Apache Airflow, Python-based)
Apache Beam (Python Based) We decided to have ArgoCD as it is simple to define the tasks and reference with a simple DSL. We will need
[x] A GKE to be deployed to GCP
[ ] ArgoCD should be deployed to this GKE
[ ] At least one container to call from the workflow

ebrucucen commented 3 years ago

[] We have deployed GKE (with default settings, may need to readjust the settings tomorrow to check the consumption vs cost) [] We have deployed ArgoCD following the documentation: https://argocd-operator-helm.readthedocs.io/en/latest/k8s/gcp.html (https endpoint throws an error, and redirect to non-https endpoint... we need to check it out)