Azure / Synapse-workspace-deployment

MIT License
27 stars 34 forks source link

Deployment taking long time for big WorkSpace Template file #106

Open sriharshagubba opened 1 month ago

sriharshagubba commented 1 month ago

Hi, Our deployment is taking about 1 hour when we try to run our WorkspaceTemplate.json(which is 40MB), we have significant number of pipelines,triggers,notebooks to deploy. How can we make sure to run faster?

keen85 commented 1 month ago

Hi @sriharshagubba, Apparently all artifacts are deployed strict sequencially and therefore it's slow. We are also annoyed by this and we are considering using an inofficial task that promisses much faster deployments (due to parallelization). But so far we don't have any experience with it.

sriharshagubba commented 1 month ago

We are running yaml on github runner. Do you know how we can configure yaml for this new fast-synapse-deploy task