search

AzBuilder / terrakube

Open source IaC Automation and Collaboration Software.
https://docs.terrakube.io
Apache License 2.0
492 stars 38 forks source link

Bitbucket connection issue Unexpected end of file from server #1165

Open MARVELMafia opened 1 month ago

MARVELMafia commented 1 month ago

Feedback

Hi, could the following error be the reason why I any 'plan' or 'plan & apply' I run via terrakube simply tells me that No changes. Your infrastructure matches the configuration.

[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.workspace.SetupWorkspaceImpl - vcsType: BITBUCKET
[threadPoolTaskExecutor-1] ERROR org.terrakube.executor.service.workspace.SetupWorkspaceImpl - Unexpected end of file from server
[threadPoolTaskExecutor-1] ERROR org.terrakube.executor.service.workspace.security.WorkspaceSecurityImpl - Generate Dex Authentication Private Token
[threadPoolTaskExecutor-1] ERROR org.terrakube.executor.service.executor.ExecutorJobImpl - /home/cnb/.terraform-spring-boot/executor/b61b9b32-3bac-449b-90f5-04248b52209e/8c558983-1648-412e-bf5f-9e5e1dc276a7/commitHash.info (No such file or directory)
[threadPoolTaskExecutor-1] INFO org.terrakube.client.dex.DexCredentialAuthentication - Authentication error 401
alfespa17 commented 1 month ago

I don't think this is related to dex, the main issue I think is the following Unexpected end of file from server when terrakube is trying to clone the repository

alfespa17 commented 1 month ago

Try connecting to the database directly, look for the Vcs table, update the vcs connection to PENDIG, login to the UI search the vcs connection and click the connect button and try to run your workspace again.

MARVELMafia commented 1 month ago

Try connecting to the database directly, look for the Vcs table, update the vcs connection to PENDIG, login to the UI search the vcs connection and click the connect button and try to run your workspace again.

@alfespa17 thanks for your reply. What database are you talking about? and do you mean PENDIG or PENDING? my connection via the UI to the bitbucket repo seems fine image

alfespa17 commented 1 month ago

Try connecting to the database directly, look for the Vcs table, update the vcs connection to PENDIG, login to the UI search the vcs connection and click the connect button and try to run your workspace again.

@alfespa17 thanks for your reply. What database are you talking about? and do you mean PENDIG or PENDING? my connection via the UI to the bitbucket repo seems fine image

It looks good in the UI but maybe the internal refresh token failed, so forcing the vcs connection might fix your issue. Try connection to the terrakube database and change the values that I mentioned above

alfespa17 commented 1 month ago

This is the default password and you can see the user in the same place

https://github.com/AzBuilder/terrakube-helm-chart/blob/6b6dbcd51cc34e7ff92ed63df491babc605cbb6b/charts/terrakube/values.yaml#L161

MARVELMafia commented 1 month ago

This is the default password and you can see the user in the same place

https://github.com/AzBuilder/terrakube-helm-chart/blob/6b6dbcd51cc34e7ff92ed63df491babc605cbb6b/charts/terrakube/values.yaml#L161

Thanks. I maanaged to log in as user postgres. I updated the table: image

but now I do not see my organization in the UI it seems to have gotten reset completely

2024-08-06 12:16:25.807 ERROR 1 --- [nio-8080-exec-9] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.RuntimeException: java.lang.IllegalArgumentException: No enum constant org.terrakube.api.rs.vcs.VcsStatus.PENDIG] with root cause

java.lang.IllegalArgumentException: No enum constant org.terrakube.api.rs.vcs.VcsStatus.PENDIG
        at java.base/java.lang.Enum.valueOf(Unknown Source) ~[na:na]
        at org.hibernate.type.descriptor.java.EnumJavaTypeDescriptor.fromName(EnumJavaTypeDescriptor.java:84) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.type.descriptor.java.EnumJavaTypeDescriptor.wrap(EnumJavaTypeDescriptor.java:54) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.type.descriptor.java.EnumJavaTypeDescriptor.wrap(EnumJavaTypeDescriptor.java:16) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.type.descriptor.sql.VarcharTypeDescriptor$2.doExtract(VarcharTypeDescriptor.java:62) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.type.descriptor.sql.BasicExtractor.extract(BasicExtractor.java:47) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.metamodel.model.convert.internal.NamedEnumValueConverter.readValue(NamedEnumValueConverter.java:68) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.type.EnumType.nullSafeGet(EnumType.java:250) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.type.CustomType.nullSafeGet(CustomType.java:119) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.type.AbstractType.hydrate(AbstractType.java:91) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.persister.entity.AbstractEntityPersister.hydrate(AbstractEntityPersister.java:3214) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.persister.entity.Loadable.hydrate(Loadable.java:94) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.loader.plan.exec.process.internal.EntityReferenceInitializerImpl.loadFromResultSet(EntityReferenceInitializerImpl.java:342) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.loader.plan.exec.process.internal.EntityReferenceInitializerImpl.hydrateEntityState(EntityReferenceInitializerImpl.java:269) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.loader.plan.exec.process.internal.AbstractRowReader.readRow(AbstractRowReader.java:80) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.loader.plan.exec.process.internal.ResultSetProcessorImpl.extractRows(ResultSetProcessorImpl.java:157) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.loader.plan.exec.process.internal.ResultSetProcessorImpl.extractResults(ResultSetProcessorImpl.java:94) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:105) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.loader.collection.plan.AbstractLoadPlanBasedCollectionInitializer.initialize(AbstractLoadPlanBasedCollectionInitializer.java:87) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:705) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.event.internal.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:76) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:107) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.internal.SessionImpl.initializeCollection(SessionImpl.java:2208) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection$4.doWork(AbstractPersistentCollection.java:595) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:264) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:591) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection.read(AbstractPersistentCollection.java:149) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection$1.doWork(AbstractPersistentCollection.java:178) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection$1.doWork(AbstractPersistentCollection.java:163) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:264) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:162) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at org.hibernate.collection.internal.PersistentBag.size(PersistentBag.java:371) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        at com.yahoo.elide.datastores.jpql.JPQLTransaction.addSingleElement(JPQLTransaction.java:253) ~[elide-datastore-hibernate-6.1.12.jar:na]
        at com.yahoo.elide.datastores.jpql.JPQLTransaction.getToManyRelation(JPQLTransaction.java:197) ~[elide-datastore-hibernate-6.1.12.jar:na]
        at com.yahoo.elide.core.datastore.inmemory.InMemoryStoreTransaction.lambda$getToManyRelation$1(InMemoryStoreTransaction.java:84) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.core.datastore.inmemory.InMemoryStoreTransaction.fetchData(InMemoryStoreTransaction.java:262) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.core.datastore.inmemory.InMemoryStoreTransaction.getToManyRelation(InMemoryStoreTransaction.java:98) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.core.PersistentResource.getRelationUnchecked(PersistentResource.java:1340) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.core.PersistentResource.getRelation(PersistentResource.java:1260) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.core.PersistentResource.getRelationCheckedFiltered(PersistentResource.java:1218) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.core.PersistentResource.lambda$getRelationships$21(PersistentResource.java:1592) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.core.PersistentResource.getRelationshipsWithRelationshipFunction(PersistentResource.java:1629) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.core.PersistentResource.getRelationships(PersistentResource.java:1588) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.core.PersistentResource.toResource(PersistentResource.java:1538) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.core.PersistentResource.toResource(PersistentResource.java:1517) ~[elide-core-6.1.12.jar:na]
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[na:na]
        at java.base/java.util.Iterator.forEachRemaining(Unknown Source) ~[na:na]
        at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source) ~[na:na]
        at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[na:na]
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[na:na]
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[na:na]
        at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[na:na]
        at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source) ~[na:na]
        at com.yahoo.elide.jsonapi.parser.state.CollectionTerminalState.getData(CollectionTerminalState.java:158) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.jsonapi.parser.state.CollectionTerminalState.handleGet(CollectionTerminalState.java:79) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.jsonapi.parser.state.StateContext.handleGet(StateContext.java:107) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.jsonapi.parser.GetVisitor.visitQuery(GetVisitor.java:31) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.jsonapi.parser.GetVisitor.visitQuery(GetVisitor.java:18) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.generated.parsers.CoreParser$QueryContext.accept(CoreParser.java:568) ~[elide-core-6.1.12.jar:na]
        at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:46) ~[antlr4-runtime-4.9.3.jar:4.9.3]
        at com.yahoo.elide.generated.parsers.CoreBaseVisitor.visitStart(CoreBaseVisitor.java:20) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.jsonapi.parser.BaseVisitor.visitStart(BaseVisitor.java:47) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.jsonapi.parser.BaseVisitor.visitStart(BaseVisitor.java:33) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.generated.parsers.CoreParser$StartContext.accept(CoreParser.java:117) ~[elide-core-6.1.12.jar:na]
        at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18) ~[antlr4-runtime-4.9.3.jar:4.9.3]
        at com.yahoo.elide.Elide.visit(Elide.java:486) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.Elide.lambda$get$1(Elide.java:265) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.Elide.handleRequest(Elide.java:510) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.Elide.get(Elide.java:258) ~[elide-core-6.1.12.jar:na]
        at com.yahoo.elide.spring.controllers.JsonApiController$1.call(JsonApiController.java:91) ~[elide-spring-boot-autoconfigure-6.1.12.jar:na]
        at com.yahoo.elide.spring.controllers.JsonApiController$1.call(JsonApiController.java:88) ~[elide-spring-boot-autoconfigure-6.1.12.jar:na]
        at org.springframework.web.context.request.async.WebAsyncManager.lambda$startCallableProcessing$4(WebAsyncManager.java:337) ~[spring-web-5.3.31.jar:5.3.31]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
alfespa17 commented 1 month ago

It is PENDING, not PENDIG

https://github.com/AzBuilder/terrakube/blob/f30cf124b2ee03c9b719b672a5071323edb6a165/api/src/main/java/org/terrakube/api/rs/vcs/VcsStatus.java#L4

MARVELMafia commented 1 month ago

Updated to 'PENDING' then I reconnected via the UI to the VCS Provider. And then I ran my workspace again. But am getting the same error?

[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.workspace.SetupWorkspaceImpl - vcsType: BITBUCKET
[threadPoolTaskExecutor-1] ERROR org.terrakube.executor.service.workspace.SetupWorkspaceImpl - Unexpected end of file from server
[threadPoolTaskExecutor-1] ERROR org.terrakube.executor.service.workspace.security.WorkspaceSecurityImpl - Generate Dex Authentication Private Token
[threadPoolTaskExecutor-1] ERROR org.terrakube.executor.service.executor.ExecutorJobImpl - /home/cnb/.terraform-spring-boot/executor/771dfbb2-2315-4733-b4ef-91fd30de3f1b/f5e6c493-0e75-4a0c-a443-574217d023d3/commitHash.info (No such file or directory)
MARVELMafia commented 1 month ago

Updated to 'PENDING' then I reconnected via the UI to the VCS Provider. And then I ran my workspace again. But am getting the same error?

[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.workspace.SetupWorkspaceImpl - vcsType: BITBUCKET
[threadPoolTaskExecutor-1] ERROR org.terrakube.executor.service.workspace.SetupWorkspaceImpl - Unexpected end of file from server
[threadPoolTaskExecutor-1] ERROR org.terrakube.executor.service.workspace.security.WorkspaceSecurityImpl - Generate Dex Authentication Private Token
[threadPoolTaskExecutor-1] ERROR org.terrakube.executor.service.executor.ExecutorJobImpl - /home/cnb/.terraform-spring-boot/executor/771dfbb2-2315-4733-b4ef-91fd30de3f1b/f5e6c493-0e75-4a0c-a443-574217d023d3/commitHash.info (No such file or directory)

This is how it is on the UI:

image

alfespa17 commented 1 month ago

Not sure what could be the issue but I guess is related to some network connectivity.

Did you try deleting the executor pod? Can you check your workspace variables and make sure you update the values?

From the above message it looks like it not even downloading any terraform plugins, it like it is running inside an empty folder.

Can you try running this template that show the current files inside the working directory

flow:
  - type: "terraformPlan"
    step: 100
    commands:
      - runtime: "BASH"
        priority: 100
        before: true
        script: |
          pwd
          ls -al
          cd $workingDirectory
          ls -al
MARVELMafia commented 1 month ago

I know. It is really strange. I checked the working directory.

I tried again after deleteing the executor pod after updating the vcs connection.

Also checked my workspace variables but I've added global variables to the workspace and I would get 'authentication' error if any variables were wrong/not updated.

[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.workspace.SetupWorkspaceImpl - vcsType: BITBUCKET
[threadPoolTaskExecutor-1] ERROR org.terrakube.executor.service.workspace.SetupWorkspaceImpl - Unexpected end of file from server
[threadPoolTaskExecutor-1] ERROR org.terrakube.executor.service.workspace.security.WorkspaceSecurityImpl - Generate Dex Authentication Private Token
[threadPoolTaskExecutor-1] ERROR org.terrakube.executor.service.executor.ExecutorJobImpl - /home/cnb/.terraform-spring-boot/executor/60d1fd68-f8bb-40cc-9a39-f0b5391d0ad4/2473d22b-1423-45aa-b925-71420efa0a55/commitHash.info (No such file or directory)
[threadPoolTaskExecutor-1] INFO org.terrakube.client.dex.DexCredentialAuthentication - Authentication error 401
[threadPoolTaskExecutor-1] ERROR org.terrakube.client.dex.DexCredentialAuthentication - Generate Dex Authentication Private Token
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.executor.ExecutorJobImpl - Execute Plan for Organization 60d1fd68-f8bb-40cc-9a39-f0b5391d0ad4 Workspace 2473d22b-1423-45aa-b925-71420efa0a55 
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - Terraform Working Directory: /home/cnb/.terraform-spring-boot/executor/60d1fd68-f8bb-40cc-9a39-f0b5391d0ad4/2473d22b-1423-45aa-b925-71420efa0a55
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - ***************************************
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Initializing Terrakube Job 74 Step 4ee350fa-7c92-4c01-aa96-56f533180745
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Running Terraform 1.4.5
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 

***************************************
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Running Terraform Init: 
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.plugin.tfstate.aws.AwsTerraformStateImpl - Generating backend override file for terraform 1.4.5
[threadPoolTaskExecutor-1] WARN org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - Not using any SSH key to download modules
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformClient - Creating terraform downloader using custom terraform release URL: https://releases.hashicorp.com/terraform/index.json
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Initialize TerraformDownloader using custom URL
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - User Home Directory: /home/cnb
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Validate/Create download temp directory: /home/cnb/.terraform-spring-boot/download/
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Validate/Create terraform directory: /home/cnb/.terraform-spring-boot/terraform/
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - User Home Directory for tofu download: /home/cnb
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Validate/Create tofu download temp directory: /home/cnb/.terraform-spring-boot/download/tofu/
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Validate/Create tofu directory: /home/cnb/.terraform-spring-boot/tofu/
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Downloading terraform releases list
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Deleting Temp /home/cnb/.terraform-spring-boot/terraform-15267347612654156475-release
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Found 333 terraform releases
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Downloading terraform version 1.4.5 architecture amd64 Type Linux
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Downloading: https://releases.hashicorp.com/terraform/1.4.5/terraform_1.4.5_linux_amd64.zip
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Unzip: /home/cnb/.terraform-spring-boot/terraform/1.4.5/terraform
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Terraform setExecutable successful
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Initializing the backend...
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Successfully configured the backend "s3"! Terraform will automatically
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - use this backend unless the backend configuration changes.
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Initializing provider plugins...
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Terraform has been successfully initialized!
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - ***************************************
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Command Priority 100
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Running BASH
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Script Information:  
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - pwd
ls -al
cd $workingDirectory
ls -al
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 

***************************************
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Begin Execution: 
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.scripts.bash.BashEngine - ScriptPath: file:/home/cnb/.terraform-spring-boot/executor/60d1fd68-f8bb-40cc-9a39-f0b5391d0ad4/2473d22b-1423-45aa-b925-71420efa0a55/userScript.sh
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.scripts.bash.BashEngine - Job Complete Path :/home/cnb/.terraform-spring-boot/executor/60d1fd68-f8bb-40cc-9a39-f0b5391d0ad4/2473d22b-1423-45aa-b925-71420efa0a55/.terrakube/toolsRepository/bash/helloWorld:/home/cnb/.terraform-spring-boot/executor/60d1fd68-f8bb-40cc-9a39-f0b5391d0ad4/2473d22b-1423-45aa-b925-71420efa0a55/.terrakube/toolsRepository/bash/terratag:/home/cnb/.terraform-spring-boot/executor/60d1fd68-f8bb-40cc-9a39-f0b5391d0ad4/2473d22b-1423-45aa-b925-71420efa0a55/.terrakube/toolsRepository/bash/helloWorld:/home/cnb/.terraform-spring-boot/executor/60d1fd68-f8bb-40cc-9a39-f0b5391d0ad4/2473d22b-1423-45aa-b925-71420efa0a55/.terrakube/toolsRepository/bash/terratag:/home/cnb/.terraform-spring-boot/executor/60d1fd68-f8bb-40cc-9a39-f0b5391d0ad4/2473d22b-1423-45aa-b925-71420efa0a55:/home/cnb/.terraform-spring-boot/terraform/1.4.5
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - /home/cnb/.terraform-spring-boot/executor/60d1fd68-f8bb-40cc-9a39-f0b5391d0ad4/2473d22b-1423-45aa-b925-71420efa0a55
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - total 8
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - drwxr-xr-x 4 cnb cnb 116 Aug  7 09:39 .
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - drwxr-xr-x 3 cnb cnb  58 Aug  7 09:39 ..
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - drwxr-xr-x 2 cnb cnb  39 Aug  7 09:39 .terraform
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.scripts.bash.BashEngine - Exit code 0
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - End Execution... 
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - drwxr-xr-x 3 cnb cnb  37 Aug  7 09:39 .terrakube
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - -rw-r--r-- 1 cnb cnb 534 Aug  7 09:39 azure_backend_override.tf
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - -rwxr--r-- 1 cnb cnb  38 Aug  7 09:39 userScript.sh
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 

[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - total 8
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - drwxr-xr-x 4 cnb cnb 116 Aug  7 09:39 .
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - drwxr-xr-x 3 cnb cnb  58 Aug  7 09:39 ..
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - drwxr-xr-x 2 cnb cnb  39 Aug  7 09:39 .terraform
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - drwxr-xr-x 3 cnb cnb  37 Aug  7 09:39 .terrakube
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - -rw-r--r-- 1 cnb cnb 534 Aug  7 09:39 azure_backend_override.tf
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - -rwxr--r-- 1 cnb cnb  38 Aug  7 09:39 userScript.sh
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - ***************************************
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Running Terraform PLAN
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - ***************************************
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.plugin.tfstate.aws.AwsTerraformStateImpl - Generating backend override file for terraform 1.4.5
[threadPoolTaskExecutor-1] WARN org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - Not using any SSH key to download modules
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformClient - Creating terraform downloader using custom terraform release URL: https://releases.hashicorp.com/terraform/index.json
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Initialize TerraformDownloader using custom URL
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - User Home Directory: /home/cnb
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Validate/Create download temp directory: /home/cnb/.terraform-spring-boot/download/
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Validate/Create terraform directory: /home/cnb/.terraform-spring-boot/terraform/
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - User Home Directory for tofu download: /home/cnb
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Validate/Create tofu download temp directory: /home/cnb/.terraform-spring-boot/download/tofu/
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Validate/Create tofu directory: /home/cnb/.terraform-spring-boot/tofu/
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Downloading terraform releases list
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Deleting Temp /home/cnb/.terraform-spring-boot/terraform-1161960951654714461-release
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Found 333 terraform releases
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Downloading terraform version 1.4.5 architecture amd64 Type Linux
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - terraform_1.4.5_linux_amd64.zip already exists
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - No changes. Your infrastructure matches the configuration.
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Terraform has compared your real infrastructure against your configuration
alfespa17 commented 1 month ago

Yeah, is weird @MARVELMafia, what I can see from your logs is that there are no files downloaded from bitbucket.

I can only see 2 folders that are used by terrakube called .terraform, .terrakube and the azure_backend_override.tf (that is a typo error when creating the internal backend override file here that is not related to your issue)

[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - /home/cnb/.terraform-spring-boot/executor/60d1fd68-f8bb-40cc-9a39-f0b5391d0ad4/2473d22b-1423-45aa-b925-71420efa0a55
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - total 8
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - drwxr-xr-x 4 cnb cnb 116 Aug  7 09:39 .
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - drwxr-xr-x 3 cnb cnb  58 Aug  7 09:39 ..
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - drwxr-xr-x 2 cnb cnb  39 Aug  7 09:39 .terraform
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.scripts.bash.BashEngine - Exit code 0
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - End Execution... 
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - drwxr-xr-x 3 cnb cnb  37 Aug  7 09:39 .terrakube
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - -rw-r--r-- 1 cnb cnb 534 Aug  7 09:39 azure_backend_override.tf
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - -rwxr--r-- 1 cnb cnb  38 Aug  7 09:39 userScript.sh
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 

[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - total 8
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - drwxr-xr-x 4 cnb cnb 116 Aug  7 09:39 .
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - drwxr-xr-x 3 cnb cnb  58 Aug  7 09:39 ..
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - drwxr-xr-x 2 cnb cnb  39 Aug  7 09:39 .terraform
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - drwxr-xr-x 3 cnb cnb  37 Aug  7 09:39 .terrakube
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - -rw-r--r-- 1 cnb cnb 534 Aug  7 09:39 azure_backend_override.tf
[ForkJoinPool-2-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - -rwxr--r-- 1 cnb cnb  38 Aug  7 09:39 userScript.sh
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - ***************************************
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Running Terraform PLAN
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - ***************************************

I am just guessing now maybe the following.

Did you check if the repository that you were using was renamed? Do you know if the repository has some git submodules (maybe related to this if you are using 2.22.0)? Maybe you can check the commits inside the repository or change the branch to test? double check that your oauth app still have the necesarry access permission from here can you create a simple workspace in another repository just to double check if the oauth app in bitbucket can download the code?

MARVELMafia commented 1 month ago

@alfespa17 The repository name is the same. I just used terrakube after 1 month of not using and started seeing these errors. I am using terrakube version 2.19.2 I tried created a different workspace and tried pulling the same repo but faced the same errors.

I just tried

can you create a simple workspace in another repository just to double check if the oauth app in bitbucket can download the code?

I created a new oauth consumer on bitbucket and tested it again and I see the following:

image

I feel like something is terribly rotten as no workspaces are getting cloned.

alfespa17 commented 1 month ago

Try creating a new vcs connection using a new oauth application, once created connect to the database and update the vcs_id property inside the workspace table, maybe that works for you

MARVELMafia commented 1 month ago

I just attempted to do that and I see the same errors. I think its time I park my terrakube quest. Thank you @alfespa17 for your continuous help for the past few months while I was setting this project and trying to use it in production.

ehuizar1028 commented 4 weeks ago

I am getting the same error, but I am not using bitbucket, instead I am using GitHub Enterprise. '[threadPoolTaskExecutor-1] ERROR org.terrakube.executor.service.workspace.security.WorkspaceSecurityImpl - Generate Dex Authentication Private Token'

It seems that the token expires after a certain amount of time (I'm assuming 8 hours), since the token generated by the GitHub App is set to only last 8 hours, but should refresh with the refresh token.

The refresh token in the database is set to null, so I'm assuming that the token is not getting refreshed. in the UI, i see the connect to GitHub button, and after hitting that it works.

Any guidance @alfespa17 ? Anything that will cause the token to get refreshed?

alfespa17 commented 4 weeks ago

I am getting the same error, but I am not using bitbucket, instead I am using GitHub Enterprise. '[threadPoolTaskExecutor-1] ERROR org.terrakube.executor.service.workspace.security.WorkspaceSecurityImpl - Generate Dex Authentication Private Token'

It seems that the token expires after a certain amount of time (I'm assuming 8 hours), since the token generated by the GitHub App is set to only last 8 hours, but should refresh with the refresh token.

The refresh token in the database is set to null, so I'm assuming that the token is not getting refreshed. in the UI, i see the connect to GitHub button, and after hitting that it works.

Any guidance @alfespa17 ? Anything that will cause the token to get refreshed?

Maybe it is a token expiration configuration restriction in github enterprise, as you can see here we create a task to refresh the token for other vcs provider but it for github oauth app it was not needed it when I added that feature some time ago.

https://github.com/AzBuilder/terrakube/blob/7dc6aad3896c7164f7385ea87af7124ced54f9a3/api/src/main/java/org/terrakube/api/plugin/vcs/TokenService.java#L54

ehuizar1028 commented 4 weeks ago

@alfespa17 thanks! so there's an option to disable token expiration, but we'd rather go the safer route if possible and refresh the tokens.

If we can add this to the road map for the next release, that'd be great! This product is great BTW! We are still testing it but we can't wait to fully productionize this!

Thanks!

alfespa17 commented 4 weeks ago

@alfespa17 thanks! so there's an option to disable token expiration, but we'd rather go the safer route if possible and refresh the tokens.

If we can add this to the road map for the next release, that'd be great! This product is great BTW! We are still testing it but we can't wait to fully productionize this!

Thanks!

I would like to add something like that but I don't have a github enterprise to test github oauth app with token expiration.

There is a pull request that adds support for github apps that I still need to review next week,

in the future you can use a github app (I understand that you have better security options) instead of the github oauth app that is currently supported.

ehuizar1028 commented 4 weeks ago

interesting, i am actually already using github-app as authentication, following this guide but instead creating a github-app instead of an oauth.

https://docs.terrakube.io/user-guide/vcs-providers/github-enterprise

it works fine except for the token refresh.

alfespa17 commented 4 weeks ago

interesting, i am actually already using github-app as authentication, following this guide but instead creating a github-app instead of an oauth.

https://docs.terrakube.io/user-guide/vcs-providers/github-enterprise

it works fine except for the token refresh.

GIthub oauth apps are the only supported right now, maybe for next version we add github apps.