Open rajnekkanti opened 1 month ago
alfespa17
commented
1 month ago There is a comment related to that in the documentation here
And for the helm chart maybe it will be a new feature to add some code to create the ingress if you are using the executor agent in a different cluster
rajnekkanti
commented
1 month ago Hi, @alfespa17 was able to setup self-hosted executor agent in different EKS cluster where terrakube is not running. So Terrakube runs in EKS-1 cluster and self-hosted executor agent runs in EKS-2 cluster. I'm able to run jobs from UI or CLI on this executor agent but logs or terraform plan is not showing up in UI.
if run from CLI, i can see it in terminal but not in terrakube UI.
Setup:
Any thoughts? Thanks!
alfespa17
commented
1 month ago can check if you see any error in the API logs when you are checking the logs from the UI?
rajnekkanti
commented
1 month ago I couldn't find anything abnormal except one error log
2024-08-01T15:38:29.326Z INFO 1 --- [nio-8080-exec-2] o.t.a.plugin.state.RemoteTfeController : Create new run
2024-08-01T15:38:29.326Z INFO 1 --- [nio-8080-exec-2] o.t.api.plugin.state.RemoteTfeService : Creating new Terrakube Job
2024-08-01T15:38:29.326Z INFO 1 --- [nio-8080-exec-2] o.t.api.plugin.state.RemoteTfeService : Workspace f465e8bd-dc8a-4a2f-82a6-5298aa28f76d Configuration 70e2cb1f-0448-4171-beac-ed0f1d882c18
2024-08-01T15:38:29.326Z INFO 1 --- [nio-8080-exec-2] o.t.api.plugin.state.RemoteTfeService : isDestroy false autoApply false
2024-08-01T15:38:29.335Z INFO 1 --- [nio-8080-exec-2] o.t.api.plugin.state.RemoteTfeService : Creating Job
2024-08-01T15:38:29.335Z WARN 1 --- [nio-8080-exec-2] o.t.api.plugin.state.RemoteTfeService : Workspace is using VCS connection, overriding vcs source and branch to run job using a remote plan
2024-08-01T15:38:29.337Z INFO 1 --- [nio-8080-exec-2] o.t.api.plugin.state.RemoteTfeService : Job Created
2024-08-01T15:38:29.337Z INFO 1 --- [nio-8080-exec-2] o.t.a.p.scheduler.ScheduleJobService : Create Job Context DEFAULT.TerrakubeV2_Job_714
2024-08-01T15:38:29.341Z INFO 1 --- [nio-8080-exec-2] o.t.api.plugin.state.RemoteTfeService : Job Context Created
2024-08-01T15:38:29.341Z INFO 1 --- [nio-8080-exec-2] o.t.api.plugin.state.RemoteTfeService : Searching Run 714
2024-08-01T15:38:29.344Z INFO 1 --- [nio-8080-exec-2] o.t.api.plugin.state.RemoteTfeService : Position in queue 0 for workspace WORKSPACENAME
2024-08-01T15:38:29.344Z INFO 1 --- [nio-8080-exec-2] o.t.api.plugin.state.RemoteTfeService : Included: null
2024-08-01T15:38:29.344Z INFO 1 --- [nio-8080-exec-2] o.t.api.plugin.state.RemoteTfeService : RunsData(data=RunsModel(attributes={resource-changes=1, permissions={can-apply=true}, has-changes=true, position-in-queue=0, resource-additions=1, actions={is-confirmable=true, is-discardable=true}, resource-destructions=0, status=pending}, relationships=Relationships(configurationVersion=null, workspace=org.terrakube.api.plugin.state.model.runs.WorkspaceModel@59a01cae, plan=org.terrakube.api.plugin.state.model.runs.PlanModel@7bed42, apply=org.terrakube.api.plugin.state.model.runs.ApplyModel@1b2cb541, runEventsModel=org.terrakube.api.plugin.state.model.runs.RunEventsModel@596b0f03)), included=null)
2024-08-01T15:38:30.570Z INFO 1 --- [nio-8080-exec-9] o.t.a.plugin.state.RemoteTfeController : Searching: ORGNAME WORKSPACENAME
2024-08-01T15:38:30.573Z INFO 1 --- [nio-8080-exec-9] o.t.api.plugin.state.RemoteTfeService : Found Workspace Id: f465e8bd-dc8a-4a2f-82a6-5298aa28f76d Terraform: 1.5.3
2024-08-01T15:38:30.575Z INFO 1 --- [nio-8080-exec-9] o.t.api.plugin.state.RemoteTfeService : Found Current Job Id: 714
2024-08-01T15:38:32.670Z INFO 1 --- [io-8080-exec-10] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: /tfoutput/context/713/context.json
2024-08-01T15:38:33.417Z INFO 1 --- [nio-8080-exec-3] o.t.api.plugin.state.RemoteTfeService : Searching Run 714
2024-08-01T15:38:33.425Z INFO 1 --- [nio-8080-exec-3] o.t.api.plugin.state.RemoteTfeService : Position in queue 0 for workspace WORKSPACENAME
2024-08-01T15:38:33.425Z INFO 1 --- [nio-8080-exec-3] o.t.api.plugin.state.RemoteTfeService : Included: null
2024-08-01T15:38:33.426Z INFO 1 --- [nio-8080-exec-3] o.t.api.plugin.state.RemoteTfeService : RunsData(data=RunsModel(attributes={resource-changes=1, permissions={can-apply=true}, has-changes=true, position-in-queue=0, resource-additions=1, actions={is-confirmable=true, is-discardable=true}, resource-destructions=0, status=pending}, relationships=Relationships(configurationVersion=null, workspace=org.terrakube.api.plugin.state.model.runs.WorkspaceModel@31d7bae1, plan=org.terrakube.api.plugin.state.model.runs.PlanModel@290dcf2f, apply=org.terrakube.api.plugin.state.model.runs.ApplyModel@2a83a78c, runEventsModel=org.terrakube.api.plugin.state.model.runs.RunEventsModel@3be6db18)), included=null)
2024-08-01T15:38:34.391Z INFO 1 --- [nio-8080-exec-6] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: /tfoutput/context/712/context.json
2024-08-01T15:38:35.039Z INFO 1 --- [nio-8080-exec-5] o.t.api.plugin.state.RemoteTfeService : Searching Run 714
2024-08-01T15:38:35.047Z INFO 1 --- [nio-8080-exec-5] o.t.api.plugin.state.RemoteTfeService : Position in queue 0 for workspace WORKSPACENAME
2024-08-01T15:38:35.047Z INFO 1 --- [nio-8080-exec-5] o.t.api.plugin.state.RemoteTfeService : Included: null
2024-08-01T15:38:35.047Z INFO 1 --- [nio-8080-exec-5] o.t.api.plugin.state.RemoteTfeService : RunsData(data=RunsModel(attributes={resource-changes=1, permissions={can-apply=true}, has-changes=true, position-in-queue=0, resource-additions=1, actions={is-confirmable=true, is-discardable=true}, resource-destructions=0, status=pending}, relationships=Relationships(configurationVersion=null, workspace=org.terrakube.api.plugin.state.model.runs.WorkspaceModel@4093a2de, plan=org.terrakube.api.plugin.state.model.runs.PlanModel@2d7122e3, apply=org.terrakube.api.plugin.state.model.runs.ApplyModel@6b4e257, runEventsModel=org.terrakube.api.plugin.state.model.runs.RunEventsModel@56772689)), included=null)
2024-08-01T15:38:37.749Z INFO 1 --- [nio-8080-exec-8] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: /tfoutput/context/713/context.json
2024-08-01T15:38:37.984Z INFO 1 --- [nio-8080-exec-1] o.t.a.p.s.c.TerraformOutputController : Reading output from storage
2024-08-01T15:38:37.984Z INFO 1 --- [nio-8080-exec-1] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: tfoutput/00cc49f5-4f9f-432c-b296-e74402819a5a/713/be8fdde8-1c8a-4c05-baf8-925cd5124f70.tfoutput
2024-08-01T15:38:47.768Z INFO 1 --- [nio-8080-exec-3] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: /tfoutput/context/713/context.json
2024-08-01T15:38:47.992Z INFO 1 --- [nio-8080-exec-7] o.t.a.p.s.c.TerraformOutputController : Reading output from storage
2024-08-01T15:38:47.992Z INFO 1 --- [nio-8080-exec-7] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: tfoutput/00cc49f5-4f9f-432c-b296-e74402819a5a/713/be8fdde8-1c8a-4c05-baf8-925cd5124f70.tfoutput
2024-08-01T15:38:50.447Z INFO 1 --- [io-8080-exec-10] o.t.api.plugin.state.RemoteTfeService : Searching Run 714
2024-08-01T15:38:50.456Z INFO 1 --- [io-8080-exec-10] o.t.api.plugin.state.RemoteTfeService : Position in queue 0 for workspace WORKSPACENAME
2024-08-01T15:38:50.456Z INFO 1 --- [io-8080-exec-10] o.t.api.plugin.state.RemoteTfeService : Included: null
2024-08-01T15:38:50.456Z INFO 1 --- [io-8080-exec-10] o.t.api.plugin.state.RemoteTfeService : RunsData(data=RunsModel(attributes={resource-changes=1, permissions={can-apply=true}, has-changes=true, position-in-queue=0, resource-additions=1, actions={is-confirmable=true, is-discardable=true}, resource-destructions=0, status=pending}, relationships=Relationships(configurationVersion=null, workspace=org.terrakube.api.plugin.state.model.runs.WorkspaceModel@37f6ff84, plan=org.terrakube.api.plugin.state.model.runs.PlanModel@25a7504f, apply=org.terrakube.api.plugin.state.model.runs.ApplyModel@59556db5, runEventsModel=org.terrakube.api.plugin.state.model.runs.RunEventsModel@22d6e7ac)), included=null)
2024-08-01T15:38:53.560Z INFO 1 --- [nio-8080-exec-1] o.t.api.plugin.state.RemoteTfeService : Searching Run 714
2024-08-01T15:38:53.568Z INFO 1 --- [nio-8080-exec-1] o.t.api.plugin.state.RemoteTfeService : Position in queue 0 for workspace WORKSPACENAME
2024-08-01T15:38:53.568Z INFO 1 --- [nio-8080-exec-1] o.t.api.plugin.state.RemoteTfeService : Included: null
2024-08-01T15:38:53.568Z INFO 1 --- [nio-8080-exec-1] o.t.api.plugin.state.RemoteTfeService : RunsData(data=RunsModel(attributes={resource-changes=1, permissions={can-apply=true}, has-changes=true, position-in-queue=0, resource-additions=1, actions={is-confirmable=true, is-discardable=true}, resource-destructions=0, status=pending}, relationships=Relationships(configurationVersion=null, workspace=org.terrakube.api.plugin.state.model.runs.WorkspaceModel@25469610, plan=org.terrakube.api.plugin.state.model.runs.PlanModel@39adca63, apply=org.terrakube.api.plugin.state.model.runs.ApplyModel@67131817, runEventsModel=org.terrakube.api.plugin.state.model.runs.RunEventsModel@20751ef3)), included=null)
2024-08-01T15:38:57.743Z INFO 1 --- [nio-8080-exec-7] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: /tfoutput/context/713/context.json
2024-08-01T15:38:57.963Z INFO 1 --- [nio-8080-exec-2] o.t.a.p.s.c.TerraformOutputController : Reading output from storage
2024-08-01T15:38:57.963Z INFO 1 --- [nio-8080-exec-2] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: tfoutput/00cc49f5-4f9f-432c-b296-e74402819a5a/713/be8fdde8-1c8a-4c05-baf8-925cd5124f70.tfoutput
2024-08-01T15:38:59.556Z INFO 1 --- [io-8080-exec-10] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: /tfoutput/context/712/context.json
2024-08-01T15:38:59.783Z INFO 1 --- [nio-8080-exec-3] o.t.api.plugin.state.RemoteTfeService : Searching Run 714
2024-08-01T15:38:59.791Z INFO 1 --- [nio-8080-exec-3] o.t.api.plugin.state.RemoteTfeService : Position in queue 0 for workspace WORKSPACENAME
2024-08-01T15:38:59.791Z INFO 1 --- [nio-8080-exec-3] o.t.api.plugin.state.RemoteTfeService : Included: null
2024-08-01T15:38:59.791Z INFO 1 --- [nio-8080-exec-3] o.t.api.plugin.state.RemoteTfeService : RunsData(data=RunsModel(attributes={resource-changes=1, permissions={can-apply=true}, has-changes=true, position-in-queue=0, resource-additions=1, actions={is-confirmable=true, is-discardable=true}, resource-destructions=0, status=pending}, relationships=Relationships(configurationVersion=null, workspace=org.terrakube.api.plugin.state.model.runs.WorkspaceModel@264f83e3, plan=org.terrakube.api.plugin.state.model.runs.PlanModel@591ef278, apply=org.terrakube.api.plugin.state.model.runs.ApplyModel@3b5a150, runEventsModel=org.terrakube.api.plugin.state.model.runs.RunEventsModel@3c0b3d8f)), included=null)
2024-08-01T15:39:00.037Z INFO 1 --- [ryBean_Worker-4] o.t.api.plugin.scheduler.ScheduleJob : Job 714 should be completed before Thu Aug 01 21:38:29 UTC 2024, current time Thu Aug 01 15:39:00 UTC 2024
2024-08-01T15:39:00.037Z INFO 1 --- [ryBean_Worker-4] o.t.api.plugin.scheduler.ScheduleJob : Checking Job 714 Status pending
2024-08-01T15:39:00.038Z INFO 1 --- [ryBean_Worker-4] o.t.api.plugin.scheduler.ScheduleJob : Checking previous jobs....
2024-08-01T15:39:00.039Z INFO 1 --- [ryBean_Worker-4] o.t.api.plugin.scheduler.ScheduleJob : Pending with plan changes true
2024-08-01T15:39:00.041Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.scheduler.job.tcl.TclService : InitialJobSetup 714
2024-08-01T15:39:00.044Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.scheduler.job.tcl.TclService : FlowConfig:
FlowConfig(flow=[Flow(type=terraformPlan, team=null, name=Plan, error=null, step=100, commands=null, templates=null, inputsEnv=null, inputsTerraform=null, importComands=null)])
2024-08-01T15:39:00.044Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.scheduler.job.tcl.TclService : Custom Job Setup:
FlowConfig(flow=[Flow(type=terraformPlan, team=null, name=Plan, error=null, step=100, commands=null, templates=null, inputsEnv=null, inputsTerraform=null, importComands=null)])
2024-08-01T15:39:00.044Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.scheduler.job.tcl.TclService : Creating step: Flow(type=terraformPlan, team=null, name=Plan, error=null, step=100, commands=null, templates=null, inputsEnv=null, inputsTerraform=null, importComands=null)
2024-08-01T15:39:00.044Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.scheduler.job.tcl.TclService : Step name Plan
2024-08-01T15:39:00.045Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.scheduler.job.tcl.TclService : Parent 714 Step created 0a3d6c9c-ea67-487c-b0e2-32b57c8b04a7
2024-08-01T15:39:00.047Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.scheduler.job.tcl.TclService : Pending steps 1
2024-08-01T15:39:00.048Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.scheduler.job.tcl.TclService : Next Command: 100
2024-08-01T15:39:00.048Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.scheduler.job.tcl.TclService : FlowConfig:
FlowConfig(flow=[Flow(type=terraformPlan, team=null, name=Plan, error=null, step=100, commands=null, templates=null, inputsEnv=null, inputsTerraform=null, importComands=null)])
2024-08-01T15:39:00.048Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.scheduler.job.tcl.TclService : Checking import commands in YAML
2024-08-01T15:39:00.048Z INFO 1 --- [ryBean_Worker-4] o.t.api.plugin.scheduler.ScheduleJob : Execute command: terraformPlan
null
2024-08-01T15:39:00.049Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.scheduler.job.tcl.TclService : Pending steps 1
2024-08-01T15:39:00.049Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.s.j.t.executor.ExecutorService : Pending Job: 714 WorkspaceId: f465e8bd-dc8a-4a2f-82a6-5298aa28f76d
2024-08-01T15:39:00.049Z WARN 1 --- [ryBean_Worker-4] o.t.a.p.s.j.t.executor.ExecutorService : Running default operation, enable headers
2024-08-01T15:39:00.049Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.s.j.t.executor.ExecutorService : Checking Variables
2024-08-01T15:39:00.049Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.s.j.t.executor.ExecutorService : Private Repository using SSH private key
2024-08-01T15:39:00.049Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.s.j.t.executor.ExecutorService : Loading default env variables to job
2024-08-01T15:39:00.050Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.s.j.t.executor.ExecutorService : Loading default env variables to job
2024-08-01T15:39:00.050Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.s.j.t.executor.ExecutorService : Job 714 Executor agent url: http://AGENT_NLB_DNS:8090/api/v1/terraform-rs
2024-08-01T15:39:00.052Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.s.j.t.executor.ExecutorService : Job 714 Executor agent url: http://AGENT_NLB_DNS:8090/api/v1/terraform-rs
2024-08-01T15:39:00.132Z INFO 1 --- [ryBean_Worker-4] o.t.a.p.s.j.t.executor.ExecutorService : Response Status: 202
2024-08-01T15:39:00.132Z INFO 1 --- [ryBean_Worker-4] o.t.api.plugin.scheduler.ScheduleJob : Executing Job 714 Step Id 0a3d6c9c-ea67-487c-b0e2-32b57c8b04a7
2024-08-01T15:39:00.132Z INFO 1 --- [ryBean_Worker-4] o.t.api.plugin.scheduler.ScheduleJob : Deleting Schedule Job Context TerrakubeV2_Job_714, InstanceId terrakube-api-69d9855dc5-95rmt17221339443311722133945569
2024-08-01T15:39:00.213Z INFO 1 --- [nio-8080-exec-7] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: content/70e2cb1f-0448-4171-beac-ed0f1d882c18/terraformContent.tar.gz
2024-08-01T15:39:02.667Z INFO 1 --- [nio-8080-exec-9] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: /tfoutput/context/713/context.json
2024-08-01T15:39:02.797Z INFO 1 --- [nio-8080-exec-3] o.t.a.p.s.c.TerraformOutputController : Reading output from storage
2024-08-01T15:39:02.797Z INFO 1 --- [nio-8080-exec-3] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: tfoutput/00cc49f5-4f9f-432c-b296-e74402819a5a/713/be8fdde8-1c8a-4c05-baf8-925cd5124f70.tfoutput
2024-08-01T15:39:02.888Z INFO 1 --- [ task-7035] o.t.api.rs.hooks.job.JobManageHook : JobCreateHook 714
2024-08-01T15:39:02.888Z WARN 1 --- [ task-7035] o.t.api.rs.hooks.job.JobManageHook : Skip new quartz job
2024-08-01T15:39:03.121Z INFO 1 --- [nio-8080-exec-2] o.t.api.plugin.state.RemoteTfeService : Checking logs for plan: 0a3d6c9c-ea67-487c-b0e2-32b57c8b04a7
2024-08-01T15:39:04.366Z INFO 1 --- [nio-8080-exec-7] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: /tfoutput/context/712/context.json
2024-08-01T15:39:05.510Z INFO 1 --- [nio-8080-exec-6] o.t.api.plugin.state.RemoteTfeService : Checking logs for plan: 0a3d6c9c-ea67-487c-b0e2-32b57c8b04a7
2024-08-01T15:39:07.740Z INFO 1 --- [nio-8080-exec-3] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: /tfoutput/context/713/context.json
2024-08-01T15:39:07.812Z INFO 1 --- [nio-8080-exec-5] o.t.api.plugin.state.RemoteTfeService : Checking logs for plan: 0a3d6c9c-ea67-487c-b0e2-32b57c8b04a7
2024-08-01T15:39:09.183Z INFO 1 --- [nio-8080-exec-1] o.t.api.plugin.state.RemoteTfeService : Checking logs for plan: 0a3d6c9c-ea67-487c-b0e2-32b57c8b04a7
2024-08-01T15:39:09.471Z INFO 1 --- [nio-8080-exec-6] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: /tfoutput/context/712/context.json
2024-08-01T15:39:12.468Z INFO 1 --- [nio-8080-exec-9] o.t.api.plugin.state.RemoteTfeService : Checking logs for plan: 0a3d6c9c-ea67-487c-b0e2-32b57c8b04a7
2024-08-01T15:39:12.674Z INFO 1 --- [nio-8080-exec-5] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: /tfoutput/context/713/context.json
2024-08-01T15:39:12.804Z INFO 1 --- [nio-8080-exec-4] o.t.a.p.s.c.TerraformOutputController : Reading output from storage
2024-08-01T15:39:12.804Z INFO 1 --- [nio-8080-exec-4] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: tfoutput/00cc49f5-4f9f-432c-b296-e74402819a5a/713/be8fdde8-1c8a-4c05-baf8-925cd5124f70.tfoutput
2024-08-01T15:39:17.991Z INFO 1 --- [nio-8080-exec-9] o.t.a.p.s.c.TerraformOutputController : Reading output from storage
2024-08-01T15:39:17.992Z INFO 1 --- [nio-8080-exec-9] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: tfoutput/00cc49f5-4f9f-432c-b296-e74402819a5a/713/be8fdde8-1c8a-4c05-baf8-925cd5124f70.tfoutput
2024-08-01T15:39:21.455Z INFO 1 --- [nio-8080-exec-7] o.t.api.plugin.state.RemoteTfeService : Checking logs for plan: 0a3d6c9c-ea67-487c-b0e2-32b57c8b04a7
2024-08-01T15:39:22.672Z INFO 1 --- [io-8080-exec-10] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: /tfoutput/context/713/context.json
2024-08-01T15:39:22.805Z INFO 1 --- [nio-8080-exec-6] o.t.a.p.s.c.TerraformOutputController : Reading output from storage
2024-08-01T15:39:22.805Z INFO 1 --- [nio-8080-exec-6] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: tfoutput/00cc49f5-4f9f-432c-b296-e74402819a5a/713/be8fdde8-1c8a-4c05-baf8-925cd5124f70.tfoutput
2024-08-01T15:39:24.376Z INFO 1 --- [nio-8080-exec-5] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: /tfoutput/context/712/context.json
2024-08-01T15:39:25.108Z INFO 1 --- [nio-8080-exec-9] o.t.api.plugin.state.RemoteTfeService : Checking logs for plan: 0a3d6c9c-ea67-487c-b0e2-32b57c8b04a7
2024-08-01T15:39:26.735Z INFO 1 --- [nio-8080-exec-7] o.t.api.plugin.state.RemoteTfeService : Checking logs for plan: 0a3d6c9c-ea67-487c-b0e2-32b57c8b04a7
2024-08-01T15:39:26.736Z INFO 1 --- [nio-8080-exec-7] o.t.api.plugin.state.RemoteTfeService : Data length 0
2024-08-01T15:39:26.736Z INFO 1 --- [nio-8080-exec-7] o.t.api.plugin.state.RemoteTfeService : Data length 73
2024-08-01T15:39:26.737Z INFO 1 --- [nio-8080-exec-7] o.t.api.plugin.state.RemoteTfeService : Data length 64
2024-08-01T15:39:26.737Z INFO 1 --- [nio-8080-exec-7] o.t.api.plugin.state.RemoteTfeService : Data length 23
2024-08-01T15:39:26.737Z INFO 1 --- [nio-8080-exec-7] o.t.api.plugin.state.RemoteTfeService : Data length 16
2024-08-01T15:39:26.738Z INFO 1 --- [nio-8080-exec-7] o.t.api.plugin.state.RemoteTfeService : Data length 594
...
2024-08-01T15:39:27.183Z INFO 1 --- [nio-8080-exec-2] o.t.api.plugin.state.RemoteTfeService : Checking logs for plan: 0a3d6c9c-ea67-487c-b0e2-32b57c8b04a7
2024-08-01T15:39:27.184Z INFO 1 --- [nio-8080-exec-2] o.t.api.plugin.state.RemoteTfeService : Data length 53
2024-08-01T15:39:27.185Z INFO 1 --- [nio-8080-exec-2] o.t.api.plugin.state.RemoteTfeService : Data length 70
2024-08-01T15:39:27.185Z INFO 1 --- [nio-8080-exec-2] o.t.api.plugin.state.RemoteTfeService : Data length 52
...
2024-08-01T15:39:27.322Z INFO 1 --- [nio-8080-exec-1] o.t.api.plugin.state.RemoteTfeService : Checking logs for plan: 0a3d6c9c-ea67-487c-b0e2-32b57c8b04a7
2024-08-01T15:39:27.323Z INFO 1 --- [nio-8080-exec-1] o.t.api.plugin.state.RemoteTfeService : Data length 67
2024-08-01T15:39:27.323Z INFO 1 --- [nio-8080-exec-1] o.t.api.plugin.state.RemoteTfeService : Data length 50
...
2024-08-01T15:39:27.333Z INFO 1 --- [nio-8080-exec-1] o.t.api.plugin.state.RemoteTfeService : Data length 51
2024-08-01T15:39:27.334Z INFO 1 --- [nio-8080-exec-1] o.t.api.plugin.state.RemoteTfeService : Data length 4
2024-08-01T15:39:27.421Z INFO 1 --- [nio-8080-exec-8] o.t.api.plugin.state.RemoteTfeService : Checking logs for plan: 0a3d6c9c-ea67-487c-b0e2-32b57c8b04a7
2024-08-01T15:39:29.533Z INFO 1 --- [nio-8080-exec-4] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: /tfoutput/context/712/context.json
2024-08-01T15:39:29.924Z INFO 1 --- [nio-8080-exec-7] o.t.api.plugin.state.RemoteTfeService : Checking logs for plan: 0a3d6c9c-ea67-487c-b0e2-32b57c8b04a7
2024-08-01T15:39:32.204Z INFO 1 --- [nio-8080-exec-6] o.t.api.plugin.state.RemoteTfeService : Checking logs for plan: 0a3d6c9c-ea67-487c-b0e2-32b57c8b04a7
2024-08-01T15:39:34.355Z INFO 1 --- [nio-8080-exec-9] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: /tfoutput/context/712/context.json
2024-08-01T15:39:35.091Z INFO 1 --- [nio-8080-exec-2] o.t.api.plugin.state.RemoteTfeService : Checking logs for plan: 0a3d6c9c-ea67-487c-b0e2-32b57c8b04a7
2024-08-01T15:39:36.969Z INFO 1 --- [ task-7042] o.t.api.rs.hooks.job.JobManageHook : JobCreateHook 714
2024-08-01T15:39:36.969Z INFO 1 --- [ task-7042] o.t.api.rs.hooks.job.JobManageHook : Creating new quartz job
2024-08-01T15:39:36.969Z INFO 1 --- [ task-7042] o.t.a.p.scheduler.ScheduleJobService : Running Job Context Now: 714, Identity: TerrakubeV2_Job_714_6b7b65ea-8ef4-483e-b192-31895da3bd71
2024-08-01T15:39:37.032Z INFO 1 --- [ryBean_Worker-5] o.t.api.plugin.scheduler.ScheduleJob : Job 714 should be completed before Thu Aug 01 21:38:29 UTC 2024, current time Thu Aug 01 15:39:37 UTC 2024
2024-08-01T15:39:37.032Z INFO 1 --- [ryBean_Worker-5] o.t.api.plugin.scheduler.ScheduleJob : Checking Job 714 Status failed
2024-08-01T15:39:37.032Z INFO 1 --- [ryBean_Worker-5] o.t.api.plugin.scheduler.ScheduleJob : Checking previous jobs....
2024-08-01T15:39:37.035Z INFO 1 --- [ryBean_Worker-5] o.t.api.plugin.scheduler.ScheduleJob : Deleting Failed/Cancelled/Rejected Job Context TerrakubeV2_Job_714 from Quartz
2024-08-01T15:39:37.035Z WARN 1 --- [ryBean_Worker-5] o.t.api.plugin.scheduler.ScheduleJob : Cancelling pending steps
2024-08-01T15:39:37.110Z ERROR 1 --- [ryBean_Worker-5] t.a.p.s.j.t.e.e.EphemeralExecutorService : Failure executing: GET at: https://172.20.0.1:443/apis/batch/v1/namespaces/terrakube/jobs/job-714. Message: jobs.batch "job-714" is forbidden: User "system:serviceaccount:terrakube:default" cannot get resource "jobs" in API group "batch" in the namespace "terrakube". Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=batch, kind=jobs, name=job-714, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=jobs.batch "job-714" is forbidden: User "system:serviceaccount:terrakube:default" cannot get resource "jobs" in API group "batch" in the namespace "terrakube", metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).
2024-08-01T15:39:53.070Z INFO 1 --- [nio-8080-exec-2] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: /tfoutput/context/713/context.json
...
2024-08-01T15:42:44.356Z INFO 1 --- [nio-8080-exec-5] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: /tfoutput/context/712/context.json
2024-08-01T15:42:46.016Z INFO 1 --- [nio-8080-exec-4] o.t.api.plugin.state.RemoteTfeService : EntitlementData(data=Resource(type=entitlement-sets, id=org-madtech))
2024-08-01T15:42:46.813Z INFO 1 --- [nio-8080-exec-9] o.t.a.plugin.state.RemoteTfeController : Creating Configuration Version for worspaceId f465e8bd-dc8a-4a2f-82a6-5298aa28f76d
2024-08-01T15:42:46.813Z INFO 1 --- [nio-8080-exec-9] o.t.api.plugin.state.RemoteTfeService : Create Configuration Version ConfigurationData(data=ConfigurationModel(attributes={auto-queue-runs=false, speculative=true}))
2024-08-01T15:42:46.813Z INFO 1 --- [nio-8080-exec-9] o.t.api.plugin.state.RemoteTfeService : Speculative true
2024-08-01T15:42:46.813Z INFO 1 --- [nio-8080-exec-9] o.t.api.plugin.state.RemoteTfeService : Auto Queue Runs false
2024-08-01T15:42:46.814Z INFO 1 --- [nio-8080-exec-9] o.t.api.plugin.state.RemoteTfeService : New content with id 1cf9d2dd-323e-4bff-a796-db0d32b66d85 saved
2024-08-01T15:42:46.814Z INFO 1 --- [nio-8080-exec-9] o.t.api.plugin.state.RemoteTfeService : upload-url https://TERRAKUBE_API_HOST/remote/tfe/v2/configuration-versions/1cf9d2dd-323e-4bff-a796-db0d32b66d85
2024-08-01T15:43:05.350Z INFO 1 --- [io-8080-exec-10] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : Searching: /tfoutput/context/712/context.json
2024-08-01T15:43:15.788Z INFO 1 --- [nio-8080-exec-6] o.t.a.plugin.state.RemoteTfeController : Create new run
2024-08-01T15:43:15.788Z INFO 1 --- [nio-8080-exec-6] o.t.api.plugin.state.RemoteTfeService : Creating new Terrakube Job
2024-08-01T15:43:15.788Z INFO 1 --- [nio-8080-exec-6] o.t.api.plugin.state.RemoteTfeService : Workspace f465e8bd-dc8a-4a2f-82a6-5298aa28f76d Configuration 1cf9d2dd-323e-4bff-a796-db0d32b66d85
Feature description 💡
Currently, the doc https://docs.terrakube.io/getting-started/deployment/self-hosted-agents is not clear if user want to setup self hosted agent in a different cluster. Maybe add a notes on exposing redis outisde the cluster if used default redis deployment. Add an ingress option for executor deployment in helm chart to expose self hosted agent in different kubernetes cluster so Terrakube Api could talk to executor.
https://github.com/AzBuilder/terrakube/issues/718 https://github.com/AzBuilder/terrakube/pull/770
Anything else?
No response