🚀 Support for Managing Agents within a Single Pool Across Multiple Environments

BigMacIT commented 4 months ago

Description

We are currently leveraging Terraform for a large-scale deployment that spans across multiple cloud providers (including Azure and GCP) as well as on-premises environments. Our current setup involves running virtual machines (VMs) in each provider/region to achieve high availability (HA). These VMs are consolidated into a single Agent Pool, allowing any agent to execute any job irrespective of the environment. This highly matrixed setup enables, for instance, an agent in Azure to deploy resources in GCP. However, we've encountered a limitation with the use of Operator, which necessitates the management of separate Agent Pools for each deployment.

The current requirement to manage separate Agent Pools for each deployment when using Operator introduces complexity and reduces the efficiency of our infrastructure management. This setup diverges from our goal of maintaining a streamlined and flexible deployment process across our various environments.

We are seeking an enhancement to the Operator functionality that would allow for the flexible management of agents within a single, existing pool across multiple environments, eliminating the need for separate pools for each deployment context. This feature would enable us to:

Operate within a unified Agent Pool, allowing agents to be efficiently managed and utilized across all our deployment environments, including cloud providers and on-premises setups, without segregating them into distinct pools based on the deployment environment. Preserve the high degree of flexibility and efficiency characteristic of our highly matrixed infrastructure, ensuring seamless operational capabilities across diverse environments without the constraints of environment-specific agent pools. Simplify our infrastructure management by reducing the complexity and administrative burden associated with maintaining multiple, environment-specific Agent Pools. This approach would significantly enhance our ability to manage our infrastructure more effectively, leveraging a single, cohesive Agent Pool that supports deployments across any environment without the limitations imposed by the current need for multiple pools.

Potential YAML Configuration

apiVersion: app.terraform.io/v1alpha2
kind: AgentPool
metadata:
  name: cluster2-pool
  namespace: terraform-cloud-operator
spec:
  organization: my-organization
  token:
    secretKeyRef:
      name: tfc-owner
      key: token
  name: my-existing-agent-pool
  create-pool: false
  agentTokens:
    - name: cluster2-pool
  agentDeployment:
    replicas: 1
    spec:
      containers:
        - name: tfc-agent
          image: "hashicorp/tfc-agent:latest"

References

Community Note

Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

alexsomesan commented 4 months ago

Hello and thanks for reaching out!

We need some more clarifications before we can consider this request. Can you please describe in more detail what you are referring to when saying "the current requirement to manage separate Agent Pools for each deployment". The operator does not include a concept of "deployment", so I am assuming this is something specific to your organisation. It would help us to have a clear representation of what you mean there. Also, are you intending to deploy the agents on Kubernetes or directly on the VMs you mentioned?

The AgentPool is subordinate to a TFC Organisation and can be reused across Projects and Workspaces. The Operator only manages agents deployed on Kubernetes, not on VMs. If you already have a mechanism in place to deploy your agents on VM as you mentioned, you don't need to manage the agent pool with the operator. You can just reference your single existing pool by name when creating Workspaces using the agentPool attribute (https://github.com/hashicorp/terraform-cloud-operator/blob/main/docs/api-reference.md#workspaceagentpool).

If however, you wish to deploy agents on Kubernetes using the Operator's AgentPool object, you may distribute the agent Pods uniformly across node groups (based on AZ, region, etc) using the topologySpreadConstraints attribute of a PodSpec. The agentDeployment.spec attribute accepts a complete PodSpec structure, including Pod Topology Spread Constraints.

This answer is based on a few assumptions since your issue description is a bit unclear. Looking forward to your clarification so we can discuss further options.

iBrandyJackson commented 3 months ago

A quick friendly check-in on this issue. We need a bit more clarification from you, @BigMacIT to proceed. If we don't hear back soon, we'll move forward with closing the issue to keep our backlog tidy. Looking forward to your clarification. Thanks!

hashicorp / terraform-cloud-operator