search

databrickslabs / cicd-templates

Manage your Databricks deployments and CI with code.
Other
202 stars 100 forks source link

Giving Access to Private pypi Repo Hosted in Azure to Databricks #72

Closed roenciso closed 3 years ago

roenciso commented 3 years ago

Hi, I am running into what is likely authentication issues when running dbx deploy --jobs=<job> --files-only --no-package I tried the solutions below: 1) In deployments.json: "libraries": [ { "pypi": {"package": "mypckg", "repo": "pckglink"} } ], 2) Excluding '--no-package' creates the deployment-result.json file under artifacts->dbx in my databricks workspace. It fails since the pypi package reference is incorrectly set up. It runs 'pip install --index-url repourl' since it splits the name as another package and this of course fails. The url has the token so this should work if we had this instead: "pypi": { "package": "package_name --index-url repourl" } . Any thoughts on "libraries": [ { "whl": "whllocation.whl" }, { "pypi": { "package": "--index-url repourl" } }, { "pypi": { "package": "package_name" } } ]

roenciso commented 3 years ago

I found that I had accidentally checked-in a requirements.txt file that had the package and index-url in different lines and after looking at the dbx deploy code, figured out how the parsing is done.