Split up pipeline s3 syncing into separate steps

mitodl / ocw-studio

Open Source Courseware authoring tool

BSD 3-Clause "New" or "Revised" License

7 stars 3 forks source link

Split up pipeline s3 syncing into separate steps #837

Open mbertrand opened 2 years ago

mbertrand commented 2 years ago

Currently the copy-s3-buckets step in concourse pipelines includes multiple s3 sync commands. The final one, which conditionally runs for ocw-www only, could be split into a separate step that uses the s3-sync resource. https://mitodl.slack.com/archives/G01JPJZQX8E/p1637763373291400?thread_ts=1637685129.280200&cid=G01JPJZQX8E

mbertrand commented 2 years ago

On second thought, this might increase the number of containers used in each pipeline build, increasing the odds of hitting the max containers reached error.