New approach to deploy Python requirement file in spark pool

[04diiguyi]

Problem Statement

• Currently, in this repo, we use script to deploy python packages after spark pool created. We could simply use bicep to do so.

Proposed Solution

• Use bicep to create spark pool first and then update the pool using python packages

Alternative Proposals

-

Additional Context

-

Next Steps

• [ ] Team consensus to proceed
• [ ] Complete Design Review Template (if applicable)
• [ ] Schedule Design Session

[ydaponte]

Hello @04diiguyi, maybe you can help me with this issue. I don't see the possibility to deploy the Spark pool within the bicep specification: https://learn.microsoft.com/en-us/azure/templates/microsoft.databricks/workspaces?pivots=deployment-language-bicep. Could you please point me to any documentation were you've seen that is possible? Thanks in advance!

[ydaponte]

@04diiguyi, could you please reply until 29th of November, so we decide if it is feasible to work on the issue or shall we abandon it? Thanks.