search

databricks / cli

Databricks CLI
Other
148 stars 56 forks source link

Upload local libraries even if they don't have artifact defined #1664

Closed andrewnester closed 3 months ago

andrewnester commented 3 months ago

Changes

Previously for all the libraries referenced in configuration DABs made sure that there is corresponding artifact section. But this is not really necessary and flexible, because local libraries might be built outside of dabs context. It also created difficult to follow logic in code where we back referenced libraries to artifacts which was difficult to fllow

This PR does 3 things:

  1. Allows all local libraries referenced in DABs config to be uploaded to remote
  2. Simplifies upload and glob references expand logic by doing this in single place
  3. Speed things up by uploading library only once and doing this in parallel

Tests

Added unit + integration tests + made sure that change is backward compatible (no changes in existing tests)

andrewnester commented 3 months ago

All integration test passed