Closed kenmyers-8451 closed 8 months ago
Another thing I've just tried was to add the profile to the target definition and remove the default specifier and this also did not work:
targets:
dev:
workspace:
profile: "dev"
# default: true
variables:
driver_instance_pool_id: "...-pool-d20su468"
tst:
workspace:
profile: "tst"
driver_instance_pool_id: "...-pool-n0tzdlm8"
Same error as before with it trying to use ...-pool-d20su468 during databricks bundle deploy -t tst
.
Log:
% databricks bundle deploy -t tst --log-level=debug
18:51:43 INFO start pid=18310 version=0.212.1 args="databricks, bundle, deploy, -t, tst, --log-level=debug"
18:51:43 DEBUG Loading bundle configuration from: /Users/{user_id}/8451/3CM/3cm-auto-etl/bundle.yml pid=18310
18:51:43 DEBUG Apply pid=18310 mutator=seq
18:51:43 DEBUG Apply pid=18310 mutator=seq mutator=scripts.preinit
18:51:43 DEBUG No script defined for preinit, skipping pid=18310 mutator=seq mutator=scripts.preinit
18:51:43 DEBUG Apply pid=18310 mutator=seq mutator=ProcessRootIncludes
18:51:43 DEBUG Apply pid=18310 mutator=seq mutator=ProcessRootIncludes mutator=seq
18:51:43 DEBUG Apply pid=18310 mutator=seq mutator=InitializeVariables
18:51:43 DEBUG Apply pid=18310 mutator=seq mutator=DefineDefaultTarget(default)
18:51:43 DEBUG Apply pid=18310 mutator=seq mutator=LoadGitDetails
18:51:43 DEBUG Apply pid=18310 mutator=SelectTarget(tst)
18:51:43 DEBUG Apply pid=18310 mutator=seq
18:51:43 DEBUG Apply pid=18310 mutator=seq mutator=initialize
18:51:43 INFO Phase: initialize pid=18310 mutator=seq mutator=initialize
18:51:43 DEBUG Apply pid=18310 mutator=seq mutator=initialize mutator=seq
18:51:43 DEBUG Apply pid=18310 mutator=seq mutator=initialize mutator=seq mutator=InitializeWorkspaceClient
18:51:43 DEBUG Loading tst profile from /Users/{user_id}/.databrickscfg pid=18310 sdk=true
18:51:43 DEBUG Apply pid=18310 mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser
18:51:43 DEBUG Loading tst profile from /Users/{user_id}/.databrickscfg pid=18310 sdk=true
18:51:44 DEBUG GET /api/2.0/preview/scim/v2/Me
< HTTP/2.0 200 OK
...
< HTTP/2.0 200 OK
< <Streaming response> pid=18310 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull sdk=true
18:51:49 INFO Local state is the same or newer, ignoring remote state pid=18310 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull
18:51:49 DEBUG Apply pid=18310 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deferred
18:51:49 DEBUG Apply pid=18310 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deferred mutator=terraform.Apply
Deploying resources...
18:51:51 ERROR Error: terraform apply: exit status 1
Error: cannot create job: You need the 'attach' permission on instance pool ...-pool-d20su468 to configure a job to run on it.
with databricks_job.fact_marketshare,
on bundle.tf.json line 292, in resource.databricks_job.fact_marketshare:
292: }
A member of our team sort of figured this out. This might actually be an issue with using a variable called instance_pool_id
. I didn't show all of the variables before but there was an additional variable:
targets:
dev:
default: true
variables:
instance_pool_id: "...-pool-oq2hxn6o"
driver_instance_pool_id: "...-pool-d20su468"
tst:
variables:
instance_pool_id: "...-pool-ewvowkgg"
driver_instance_pool_id: "...-pool-n0tzdlm8"
instance_pool_id
is spot-type and used for the workers when defining the cluster and driver_instance_pool_id
non-spot-type and is used for the drivers when defining a cluster.
We'd noticed in the past that actually instance_pool_id
was getting ignored and seemed to get replaced with driver_instance_pool_id
. What my teammate did was change the name instance_pool_id
to worker_instance_pool_id
and suddenly it worked. What might have actually been causing this error is that the dev driver_instance_pool_id
was being slotted into tst instance_pool_id
as well (since we see "...-pool-d20su468" as the error on tst). New targets look like this:
targets:
dev:
default: true
variables:
worker_instance_pool_id: "...-pool-oq2hxn6o"
driver_instance_pool_id: "...-pool-d20su468"
tst:
variables:
worker_instance_pool_id: "...-pool-ewvowkgg"
driver_instance_pool_id: "...-pool-n0tzdlm8"
So perhaps instance_pool_id
is some sort of protected variable or causing a scope issue?
Regardless, databricks bundle deploy -p tst -t tst
is working now with this one refactor. I will let you close this unless you want to investigate this new issue. (Also I thought about reporting this issue before – about the instance_pool_id
being ignored/replaced – but it wasn't breaking our builds like this was so it wasn't high on our priorities).
Closing this issue because we figured out the issue was on our company's end with a process overwriting the variable after we set it
Describe the issue
Hi I think this is an issue with CLI but it is related to DAB (I don't think the issue is on the DAB side, but sorry if I mislabeled this).
I'm trying to deploy a bundle from local using
databricks bundle deploy -p tst -t tst
which I expected to use the tst credentials defined in my~/.databrickscfg
and my tst target defined in bundle.yaml (I know I could update this name, my team hasn't gotten around to it). This did not work however. I got the error:However, this is the instance pool variable I defined for the dev target. Here is an excerpt of my targets:
So it seems like despite specifying to use the tst profile, it tried to use the default dev target variables (it's like -t was ignored when -p was specified).
If I run
databricks bundle deploy -t tst
without specifying the profile, I get this error:So in this case it seems like it is able to find the right target variable, but it is probably trying to do this with the dev profile since those are the default credentials.
Steps to reproduce the behavior
My .databrickscfg file looks like this
I know the tst token and credentials are correct because I can use other databricks-cli commands with
-p tst
In our bundle, the driver_instance_pool_id variables are being used in a job cluster definition, I won't share all that as I expect you test this with other variables.
Expected Behavior
You should be able to use the -p and -t parameters together during
databricks build deploy
(and maybe other commands) but you can't.Actual Behavior
Target definition, -t, seems to get ignored.
OS and CLI version
Databricks CLI v0.212.1
Is this a regression?
N/A
Debug Logs
I can share more if needed. You can see here that it seems to
SelectTarget(tst)
andLoading tst profile
but despite this it errors out with the dev target variable