iterative / terraform-provider-iterative

☁️ Terraform plugin for machine learning workloads: spot instance recovery & auto-termination | AWS, GCP, Azure, Kubernetes
https://registry.terraform.io/providers/iterative/iterative/latest/docs
Apache License 2.0
290 stars 27 forks source link

Data sync: allow more advanced `workdir` and `output` specs #590

Closed dberenbaum closed 2 years ago

dberenbaum commented 2 years ago

It would be great for workdir/output to support:

0x2b3bfa0 commented 2 years ago

Storage syntax can be easily extended to support rclone filters to include and exclude several paths or globs from data transfers:

dacbd commented 2 years ago

+1 for a exclude option,

storage {
  workdir = "."
  exclude = [
    ".dvc/cache",
    "venv"
  ]
  output = "./reports"
}
casperdcl commented 2 years ago

Quick note: exclude should be a list[str] relative to workdir