search

AzBuilder / terrakube

Open source IaC Automation and Collaboration Software.
https://docs.terrakube.io
Apache License 2.0
523 stars 44 forks source link

Rate limited by Github API #1239

Closed Foxite closed 2 months ago

Foxite commented 2 months ago

Bug description 🐞

We were running a lot of jobs on our fresh Terrakube workspaces, and suddenly the executor stopped working and reported this error: Server returned HTTP response code: 403 for URL: https://api.github.com/repos/opentofu/opentofu/releases

Followed by:

[threadPoolTaskExecutor-1] ERROR org.springframework.aop.interceptor.SimpleAsyncUncaughtExceptionHandler - Unexpected exception occurred invoking async method: public void org.terrakube.executor.service.executor.ExecutorJobImpl.createJob(org.terrakube.executor.service.mode.TerraformJob)
java.lang.NullPointerException: Cannot invoke "java.util.List.stream()" because "this.tofuReleases" is null

We discovered that a rate limit of 60 requests per hour is enforced for unauthenticated requests to the Github API.

The full logs for the job:

executor-1  | 2024-08-28T13:07:00.935059320Z Aug 28, 2024 1:07:00 PM org.apache.catalina.core.ApplicationContext log
executor-1  | 2024-08-28T13:07:00.935131529Z INFO: Initializing Spring DispatcherServlet 'dispatcherServlet'
executor-1  | 2024-08-28T13:07:00.938687680Z [http-nio-8090-exec-1] INFO org.springframework.web.servlet.DispatcherServlet - Initializing Servlet 'dispatcherServlet'
executor-1  | 2024-08-28T13:07:00.950779794Z [http-nio-8090-exec-1] INFO org.springframework.web.servlet.DispatcherServlet - Completed initialization in 12 ms
executor-1  | 2024-08-28T13:07:01.412867062Z [threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.executor.ExecutorJobImpl - Create Job for Organization 9c40287f-540f-4abc-ade4-e4a45bcff3c6 Workspace 2310cdff-5590-4eb3-b896-f1bbbb615a49 
executor-1  | 2024-08-28T13:07:01.426960072Z [threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.workspace.SetupWorkspaceImpl - User Home Directory: /home/cnb
executor-1  | 2024-08-28T13:07:02.063461537Z [threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.workspace.SetupWorkspaceImpl - Workspace git clone directory: /home/cnb/.terraform-spring-boot/executor/9c40287f-540f-4abc-ade4-e4a45bcff3c6/2310cdff-5590-4eb3-b896-f1bbbb615a49
executor-1  | 2024-08-28T13:07:02.110826213Z [threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.workspace.SetupWorkspaceImpl - vcsType: GITHUB
executor-1  | 2024-08-28T13:07:06.532594659Z [threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.workspace.SetupWorkspaceImpl - Commit Id: 5d2dc2fca17b321d59f9594a0a801684930203d9
executor-1  | 2024-08-28T13:07:06.533916520Z [threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.workspace.SetupWorkspaceImpl - Writing commit id to /home/cnb/.terraform-spring-boot/executor/9c40287f-540f-4abc-ade4-e4a45bcff3c6/2310cdff-5590-4eb3-b896-f1bbbb615a49/commitHash.info
executor-1  | 2024-08-28T13:07:06.538594850Z [threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.workspace.SetupWorkspaceImpl - Git clone: https://github.com/Vorteq/amna-platform Branch: feature/authentik-tf Folder /home/cnb/.terraform-spring-boot/executor/9c40287f-540f-4abc-ade4-e4a45bcff3c6/2310cdff-5590-4eb3-b896-f1bbbb615a49
executor-1  | 2024-08-28T13:07:06.539158977Z [threadPoolTaskExecutor-1] ERROR org.terrakube.executor.service.workspace.security.WorkspaceSecurityImpl - Generate Dex Authentication Private Token
executor-1  | 2024-08-28T13:07:06.689147977Z [threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.workspace.SetupWorkspaceImpl - Executor WorkingDir: /home/cnb/.terraform-spring-boot/executor/9c40287f-540f-4abc-ade4-e4a45bcff3c6/2310cdff-5590-4eb3-b896-f1bbbb615a49
executor-1  | 2024-08-28T13:07:06.975189710Z [threadPoolTaskExecutor-1] INFO org.terrakube.client.dex.DexCredentialAuthentication - Authentication error 401
executor-1  | 2024-08-28T13:07:06.975840886Z [threadPoolTaskExecutor-1] ERROR org.terrakube.client.dex.DexCredentialAuthentication - Generate Dex Authentication Private Token
executor-1  | 2024-08-28T13:07:07.166319211Z [threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.status.UpdateJobStatusImpl - Step list is not empty...
executor-1  | 2024-08-28T13:07:07.504701583Z [threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.executor.ExecutorJobImpl - Execute Plan for Organization 9c40287f-540f-4abc-ade4-e4a45bcff3c6 Workspace 2310cdff-5590-4eb3-b896-f1bbbb615a49 
executor-1  | 2024-08-28T13:07:07.871297654Z [threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - Terraform Working Directory: /home/cnb/.terraform-spring-boot/executor/9c40287f-540f-4abc-ade4-e4a45bcff3c6/2310cdff-5590-4eb3-b896-f1bbbb615a49/infrastructure/terraform/authentik
executor-1  | 2024-08-28T13:07:07.881105278Z [threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - ***************************************
executor-1  | 2024-08-28T13:07:07.924995572Z [threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Initializing Terrakube Job 36 Step 0f845899-03be-4234-9160-fb247759f120
executor-1  | 2024-08-28T13:07:07.926679956Z [threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Running Tofu 1.8.1
executor-1  | 2024-08-28T13:07:07.928232640Z [threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 
executor-1  | 2024-08-28T13:07:07.928278792Z 
executor-1  | 2024-08-28T13:07:07.928284398Z ***************************************
executor-1  | 2024-08-28T13:07:07.929796386Z [threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Running Tofu Init: 
executor-1  | 2024-08-28T13:07:07.930899266Z [threadPoolTaskExecutor-1] INFO org.terrakube.executor.plugin.tfstate.aws.AwsTerraformStateImpl - Generating backend override file for terraform 1.8.1
executor-1  | 2024-08-28T13:07:07.937404521Z [threadPoolTaskExecutor-1] WARN org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - Not using any SSH key to download modules
executor-1  | 2024-08-28T13:07:07.939940343Z [threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformClient - Creating terraform downloader using terraform release URL: https://releases.hashicorp.com/terraform/index.json and tofu release URL: https://api.github.com/repos/opentofu/opentofu/releases
executor-1  | 2024-08-28T13:07:07.940684563Z [threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Initialize TerraformDownloader using custom URL
executor-1  | 2024-08-28T13:07:07.940731749Z [threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - User Home Directory: /home/cnb
executor-1  | 2024-08-28T13:07:07.941353544Z [threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Validate/Create download temp directory: /home/cnb/.terraform-spring-boot/download/
executor-1  | 2024-08-28T13:07:07.941394214Z [threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Validate/Create terraform directory: /home/cnb/.terraform-spring-boot/terraform/
executor-1  | 2024-08-28T13:07:07.941408213Z [threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - User Home Directory for tofu download: /home/cnb
executor-1  | 2024-08-28T13:07:07.941419782Z [threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Validate/Create tofu download temp directory: /home/cnb/.terraform-spring-boot/download/tofu/
executor-1  | 2024-08-28T13:07:07.941428944Z [threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Validate/Create tofu directory: /home/cnb/.terraform-spring-boot/tofu/
executor-1  | 2024-08-28T13:07:07.941753329Z [threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Downloading terraform releases list
executor-1  | 2024-08-28T13:07:08.118980237Z [threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Deleting Temp /home/cnb/.terraform-spring-boot/terraform-6449338851972828690-release
executor-1  | 2024-08-28T13:07:08.119017813Z [threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Found 338 terraform releases
executor-1  | 2024-08-28T13:07:08.119108088Z [threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Downloading tofu releases list
executor-1  | 2024-08-28T13:07:08.188759717Z [threadPoolTaskExecutor-1] ERROR org.terrakube.terraform.TerraformDownloader - Server returned HTTP response code: 403 for URL: https://api.github.com/repos/opentofu/opentofu/releases
executor-1  | 2024-08-28T13:07:08.190875601Z [threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Downloading tofu version 1.8.1 architecture amd64 Type Linux
executor-1  | 2024-08-28T13:07:08.190915973Z [threadPoolTaskExecutor-1] ERROR org.springframework.aop.interceptor.SimpleAsyncUncaughtExceptionHandler - Unexpected exception occurred invoking async method: public void org.terrakube.executor.service.executor.ExecutorJobImpl.createJob(org.terrakube.executor.service.mode.TerraformJob)
executor-1  | 2024-08-28T13:07:08.192402722Z java.lang.NullPointerException: Cannot invoke "java.util.List.stream()" because "this.tofuReleases" is null
executor-1  | 2024-08-28T13:07:08.192914930Z    at org.terrakube.terraform.TerraformDownloader.downloadTofuVersion(TerraformDownloader.java:255)
executor-1  | 2024-08-28T13:07:08.193258302Z    at org.terrakube.terraform.TerraformClient.getTerraformLauncher(TerraformClient.java:283)
executor-1  | 2024-08-28T13:07:08.193586869Z    at org.terrakube.terraform.TerraformClient.run(TerraformClient.java:208)
executor-1  | 2024-08-28T13:07:08.193968732Z    at org.terrakube.terraform.TerraformClient.init(TerraformClient.java:104)
executor-1  | 2024-08-28T13:07:08.194333319Z    at org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl.executeTerraformInit(TerraformExecutorServiceImpl.java:398)
executor-1  | 2024-08-28T13:07:08.194922696Z    at org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl.plan(TerraformExecutorServiceImpl.java:103)
executor-1  | 2024-08-28T13:07:08.194933460Z    at org.terrakube.executor.service.executor.ExecutorJobImpl.createJob(ExecutorJobImpl.java:53)
executor-1  | 2024-08-28T13:07:08.195382171Z    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
executor-1  | 2024-08-28T13:07:08.195396661Z    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
executor-1  | 2024-08-28T13:07:08.195888495Z    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
executor-1  | 2024-08-28T13:07:08.196204744Z    at java.base/java.lang.reflect.Method.invoke(Unknown Source)
executor-1  | 2024-08-28T13:07:08.196528633Z    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
executor-1  | 2024-08-28T13:07:08.196538162Z    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
executor-1  | 2024-08-28T13:07:08.196543249Z    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
executor-1  | 2024-08-28T13:07:08.196567871Z    at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
executor-1  | 2024-08-28T13:07:08.196573014Z    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
executor-1  | 2024-08-28T13:07:08.196578830Z    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
executor-1  | 2024-08-28T13:07:08.196582901Z    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
executor-1  | 2024-08-28T13:07:08.196587334Z    at java.base/java.lang.Thread.run(Unknown Source)

Steps to reproduce

Make more than 60 unauthenticated requests to Github API within an hour, this should potentially be doable by starting more than 60 jobs within an hour (although we did much less than that)

Expected behavior

The executor should not fail to complete its job when the request fails, by using a cached version of tofu and ignoring a failed update check (possibly emitting a warning).

Alternatively, the executor should provide a mechanism for api requests to be authenticated (which raises the rate limit to 5000 requests per hour), presumably by reading a token from its configuration.

Example repository

No response

Anything else?

No response

alfespa17 commented 2 months ago

As a workaround you can copy the reponse from https://api.github.com/repos/opentofu/opentofu/releases to a custom endpoint similar to what we do with the terraform endpoint in here.

You need to add this environment variable CustomTofuReleasesUrlto the API and Executor,

By default it is using the github relases api as you can see here

https://github.com/AzBuilder/terrakube/blob/909b67dfeadbb1631720c032a84306b415af8be2/api/src/main/resources/application.properties#L160

and here

https://github.com/AzBuilder/terrakube/blob/909b67dfeadbb1631720c032a84306b415af8be2/executor/src/main/resources/application.properties#L9

alfespa17 commented 2 months ago

If you are using the helm chart it will be something like these:

api:
  env:
  - name: CustomTofuReleasesUrl
    value: https://my.custom.endopnt/opentofu/releases

executor:
  env:
  - name: CustomTofuReleasesUrl
    value: https://my.custom.endopnt/opentofu/releases

That should fix your issue