New approach to deploy Python requirement file in spark pool - Githubissues

Azure-Samples / modern-data-warehouse-dataops

DataOps for the Modern Data Warehouse on Microsoft Azure. https://aka.ms/mdw-dataops.

MIT License

570 stars 451 forks source link

New approach to deploy Python requirement file in spark pool #595

Open 04diiguyi opened 1 year ago

04diiguyi commented 1 year ago

Problem Statement

Currently, in this repo, we use script to deploy python packages after spark pool created. We could simply use bicep to do so.

Proposed Solution

Use bicep to create spark pool first and then update the pool using python packages

Alternative Proposals

-

Additional Context

-

Next Steps

[ ] Team consensus to proceed
[ ] Complete Design Review Template (if applicable)
[ ] Schedule Design Session