Workspace queue

[BenjaminDecreusefond]

Feedback

Hi !

As we plan to have many different workspaces, we would like to have different queues for each workspace to avoid excessive waiting time when running jobs. I thought that agent were the key to have different queues but it looks like they all share the same queue. I found different elements like ephemeral agents, self-hosted agents but I think those elements still share the same queue right ?

I wonder if this is due to the fact that I created my agents to all use the same url http://terrakube-executor-service:8090 ? Actually I'm not sure what is the purpose of the URL and how we can create agents that are able to run with another url than the one I provided above ?

Anyway we're a bit confused with all that and as to how we can achieve our goal ? I might have missed smth in the doc in which case I'm sorry but I didn't find mention of queues :/

Regards !

[alfespa17]

Feedback

Hi !

As we plan to have many different workspaces, we would like to have different queues for each workspace to avoid excessive waiting time when running jobs. I thought that agent were the key to have different queues but it looks like they all share the same queue. I found different elements like ephemeral agents, self-hosted agents but I think those elements still share the same queue right ?

I wonder if this is due to the fact that I created my agents to all use the same url http://terrakube-executor-service:8090 ? Actually I'm not sure what is the purpose of the URL and how we can create agents that are able to run with another url than the one I provided above ?

Anyway we're a bit confused with all that and as to how we can achieve our goal ? I might have missed smth in the doc in which case I'm sorry but I didn't find mention of queues :/

Regards !

It depends on your configuration, for example if you are using VCS driven workflow each workspace has its own job queue and once you trigger a job it will be execute it inside the executor agent that you define in your case the one in http://terrakube-executor-service:8090 or any other ephemeral agents, self-hosted that you have.

Any executor agents can execute jobs from different workspaces at the same time

With the CLI driven workflow is a little bit different because there are some validation at the organization level to run jobs in this part of the code:

https://github.com/hashicorp/terraform/blob/c754d95a156b7b7c6bdb173c9fbd7736630652ac/internal/cloud/backend_common.go#L173

https://github.com/hashicorp/terraform/blob/c754d95a156b7b7c6bdb173c9fbd7736630652ac/internal/cloud/backend_common.go#L196

https://github.com/hashicorp/terraform/blob/c754d95a156b7b7c6bdb173c9fbd7736630652ac/internal/cloud/backend_common.go#L201

So in this case using the CLI driven workflow you could say the queue is at organization level.

I think this could answer your question

If you install several agents like we explained here

https://docs.terrakube.io/getting-started/deployment/self-hosted-agents

You could check the service name using the following to find the correct hostname:

kubectl get svc -n YOUR-NAMESPACE-WHERE-EXECUTOR-IS-DEFINED

[BenjaminDecreusefond]

Hey @alfespa17 ! Thnaks for your reply !

Just to clarify the process. We are using VCS workspaces for now so I'm guessing we will need ephemeral-agents or self-hosted agents. Once we have deployed one of the two solutions above we should let the parameter below as Default and Terrakube will automatically use the ephemeral-agents that we just set up to manage queues, so we can run several jobs at the same time ? If that's the case what is the difference between creating ephemeral-agents and creating an agent through the API with /api/v1/organization/my-orga/agent/ ?

I'm a bit confused here 😓

[alfespa17]

Hey @alfespa17 ! Thnaks for your reply !

Just to clarify the process. We are using VCS workspaces for now so I'm guessing we will need ephemeral-agents or self-hosted agents. Once we have deployed one of the two solutions above we should let the parameter below as Default and Terrakube will automatically use the ephemeral-agents that we just set up to manage queues, so we can run several jobs at the same time ? If that's the case what is the difference between creating ephemeral-agents and creating an agent through the API with /api/v1/organization/my-orga/agent/ ?

I'm a bit confused here 😓

The setup is differnet if you want to use ephemeral agents, you will need to add one environment variable TERRAKUBE_ENABLE_EPHEMERAL_EXECUTOR=1 to the workspace like we explain here.

https://docs.terrakube.io/getting-started/deployment/ephemeral-agents#workspace-configuration

This one is to select the self hosted agent

You can find the code validation here

https://github.com/AzBuilder/terrakube/blob/f0144a0d5baa7c651f8e348371af2285a2733b7f/api/src/main/java/org/terrakube/api/plugin/scheduler/job/tcl/executor/ExecutorService.java#L166

https://github.com/AzBuilder/terrakube/blob/f0144a0d5baa7c651f8e348371af2285a2733b7f/api/src/main/java/org/terrakube/api/plugin/scheduler/job/tcl/executor/ExecutorService.java#L191

https://github.com/AzBuilder/terrakube/blob/f0144a0d5baa7c651f8e348371af2285a2733b7f/api/src/main/java/org/terrakube/api/plugin/scheduler/job/tcl/executor/ExecutorService.java#L172

[BenjaminDecreusefond]

@alfespa17 great thanks ! much clearer now ! So in the case of ephemeral-agents the parameter in the screenshot is useless as it will not be used by Terrakube ?

And so from my understanding ephemeral-agents allows for theoretically unlimited parallel number of jobs on different workspaces ?

[alfespa17]

@alfespa17 great thanks ! much clearer now ! So in the case of ephemeral-agents the parameter in the screenshot is useless as it will not be used by Terrakube ?

And so from my understanding ephemeral-agents allows for theoretically unlimited parallel number of jobs on different workspaces ?

Hello @BenjaminDecreusefond with ephemeral-agents Terrakube will create a new agent each time a job is trigger, this new agent will run the job and once it completes the process it will be automatically destroy

[BenjaminDecreusefond]

Thanks for your answers ! I'm clear now !

[BenjaminDecreusefond]

Hey @alfespa17 !

I tried to use the ephemeral-agent feature following the https://docs.terrakube.io/getting-started/deployment/ephemeral-agents and after trying to run a job on my workspace I had the following error on job:

[main] ERROR org.terrakube.executor.service.mode.batch.BatchModeServiceImpl - Unrecognized field "connectionType" (class org.terrakube.executor.service.mode.TerraformJob), not marked as ignorable (23 known properties: "vcsType", "branch", "moduleSshKey", "terraformVersion", "stepId", "commandList", "showHeader", "organizationId", "workspaceId", "commitId", "accessToken", "variables", "jobId", "type", "source", "environmentVariables", "refreshOnly", "rawState", "tofu", "folder", "terraformOutput", "refresh", "agentUrl"])
 at [Source: (String)"{"commandList":null,"type":"terraformPlan","organizationId":"179acb9d-5119-49db-a7b6-ed3aeca88daa","workspaceId":"97ba1866-a0c6-447c-9bca-dc2822db7e80","jobId":"326","stepId":"4b10bbb3-ec1f-40a6-905b-ce8525d71334","terraformVersion":"1.8.3","source":"https://github.com/myorg/myrepo.git","branch":"master","folder":"myfolder","vcsType":"GITHUB","connectionType":"OAUTH","refresh":true,"refreshOnly":false,"showHeader":true,"accessToken":"xxx"[truncated 898 chars]; line: 1, column: 410] (through reference chain: org.terrakube.executor.service.mode.TerraformJob["connectionType"])

The field connectionType is set to OAUTH but for some reason it is enable to recognize it ?

Also, I tried to cancel the job from the UI and to delete the job running kubectl delete job but it keeps coming back alive as below !

[alfespa17]

Try upgrading the default ephemeral agent version using the following

## API properties
api:
  image: "azbuilder/api-server"
  version: "2.23.3"
  serviceAccountName: "terrakube-api-service-account"
  env:
  - name: ExecutorEphemeralNamespace
    value: terrakube
  - name: ExecutorEphemeralImage
    value: azbuilder/executor:2.23.3
  - name: ExecutorEphemeralSecret
    value: terrakube-executor-secrets

I guess that if you check the job logs you will see that is using version 2.22.0

[BenjaminDecreusefond]

Oh yay my bad !