The data source databricks_jobs API currently raises an exception if jobs with duplicate names are deteced.
This seems a bit confusing since the Databricks API, asset bundles, and other SDKs in multiple languages allow creation of jobs with duplicate names.
This request is to expose a flag to this source that allows duplicate names, or provide another data source that allows retrieval of all job_id values in a workspace. In our use case, we only care about the list of IDs of jobs in our workspace.
Attempted Solutions
Our current workaround is to delete any jobs with duplicate names in our workspace. But we can't prevent new ones accidentally being prevented by developers by accident, given this limitation isn't in other Databricks SDKs.
AFAIK there otherwise isn't a solution using the existing provider to retrieve a list of job_id values in a workspace if the workspace has jobs with duplicated names.
Proposal
Given that the return value of the current databricks_jobs data source is a map of name -> Id, a new data source might be a better option to ensure backwards compatibility of the existing data source, otherwise the key/value structure of the existing map would need to be reversed.
It'd be similar to the existing implementation except the map would use the Id as a key and remove the exception. If this request is something that the maintainers would be happy with, I'd be happy to raise a PR.
Use-cases
The data source databricks_jobs API currently raises an exception if jobs with duplicate names are deteced.
This seems a bit confusing since the Databricks API, asset bundles, and other SDKs in multiple languages allow creation of jobs with duplicate names.
This request is to expose a flag to this source that allows duplicate names, or provide another data source that allows retrieval of all
job_idvalues in a workspace. In our use case, we only care about the list of IDs of jobs in our workspace.Attempted Solutions
Our current workaround is to delete any jobs with duplicate names in our workspace. But we can't prevent new ones accidentally being prevented by developers by accident, given this limitation isn't in other Databricks SDKs.
AFAIK there otherwise isn't a solution using the existing provider to retrieve a list of
job_idvalues in a workspace if the workspace has jobs with duplicated names.Proposal
Given that the return value of the current databricks_jobs data source is a map of
name -> Id, a new data source might be a better option to ensure backwards compatibility of the existing data source, otherwise the key/value structure of the existing map would need to be reversed.It'd be similar to the existing implementation except the map would use the
Idas a key and remove the exception. If this request is something that the maintainers would be happy with, I'd be happy to raise a PR.References
I've not found any duplicate or similar issues.