search

aws-controllers-k8s / community

AWS Controllers for Kubernetes (ACK) is a project enabling you to manage AWS services from Kubernetes
https://aws-controllers-k8s.github.io/community/
Apache License 2.0
2.39k stars 253 forks source link

Create a MQ broker in non-default vpc #1517

Open chakch opened 1 year ago

chakch commented 1 year ago

Describe the bug I'm not able to create a broker in a different VPC than the default one

Steps to reproduce Create an MQ broker using the following manifest: apiVersion: mq.services.k8s.aws/v1alpha1 kind: Broker metadata: name: my-broker spec: name: my-broker autoMinorVersionUpgrade: false deploymentMode: SINGLE_INSTANCE engineType: RabbitMQ engineVersion: "3.9.16" hostInstanceType: "mq.t3.micro" publiclyAccessible: false

storageType: string

securityGroups:

I think there is need to add VPCId ine the mq broker spec

Expected outcome create an Broker in VPC different from the default one

Environment

a-hilaly commented 1 year ago

Hi @chakch are you able reproduce this issue using CDK or aws cli? just trying to understand if this ACK related or an API behaviour

csantanapr commented 1 year ago

I'm able to create a broker in a specific vpc other than the default one. the requirement is that if you specify a security group from a vpc you also need to specify the subnet ids on that vpc.

Here is a yaml that I used to create a broker

apiVersion: mq.services.k8s.aws/v1alpha1
kind: Broker
metadata:
  name: mq-eks-workshop
spec:
  name: mq-eks-workshop
  deploymentMode: SINGLE_INSTANCE
  engineType: ActiveMQ
  engineVersion: "5.15.8"
  hostInstanceType: "mq.t3.micro"
  publiclyAccessible: false
  autoMinorVersionUpgrade: false
  users:
    - password:
        namespace: default
        name: mq-eks-workshop
        key: password
      groups: []
      consoleAccess: true
      username: admin
  subnetIDs:
  - $(VPC_PRIVATE_SUBNET_ID_0)
  securityGroupRefs:
    - from: 
        name: mq-eks-workshop

/close

ack-bot commented 1 year ago

@csantanapr: You can't close an active issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/aws-controllers-k8s/community/issues/1517#issuecomment-1317601316): >I'm able to create a broker in a specific vpc other than the default one. the requirement is that if you specify a security group from a vpc you also need to specify the subnet ids on that vpc. > >Here is a yaml that I used to create a broker >```yaml >apiVersion: mq.services.k8s.aws/v1alpha1 >kind: Broker >metadata: > name: mq-eks-workshop >spec: > name: mq-eks-workshop > deploymentMode: SINGLE_INSTANCE > engineType: ActiveMQ > engineVersion: "5.15.8" > hostInstanceType: "mq.t3.micro" > publiclyAccessible: false > autoMinorVersionUpgrade: false > users: > - password: > namespace: default > name: mq-eks-workshop > key: password > groups: [] > consoleAccess: true > username: admin > subnetIDs: > - $(VPC_PRIVATE_SUBNET_ID_0) > securityGroupRefs: > - from: > name: mq-eks-workshop >``` > > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
csantanapr commented 1 year ago

@chakch I didn't read closely the first time. I was able to create a ActiveMQ specifying subnet and security-group from a non default vpc and worked, but now I tried to create a RabbitMQ and I can't create it on a none-default vpc

@A-Hilaly here is a yaml with specifying subnet and security-group for rabbitmq

apiVersion: mq.services.k8s.aws/v1alpha1
kind: Broker
metadata:
  name: mq-eks-workshop
spec:
  name: mq-eks-workshop
  deploymentMode: SINGLE_INSTANCE
  engineType: RabbitMQ
  engineVersion: "3.10.10"
  hostInstanceType: "mq.t3.micro"
  publiclyAccessible: false
  autoMinorVersionUpgrade: false
  users:
    - password:
        namespace: default
        name: mq-eks-workshop
        key: password
      groups: []
      consoleAccess: true
      username: admin
  subnetIDs:
    - subnet-0e95abf6cd1647262
  securityGroups:
    - sg-0a97c3ca774be1446

The broker will be created fine, you can see it in the console with the in Running state with correct subnet and security group.

In the status of the broker you will see the error

status:
  ackResourceMetadata:
    ownerAccountID: "123456789"
    region: us-east-1
  conditions:
  - message: |-
      BadRequestException: Subnets and security groups must belong to the default VPC when only one is provided.
      {
        RespMetadata: {
          StatusCode: 400,
          RequestID: "5ffd35ab-3d65-4776-ade4-cf126af0c997"
        },
        ErrorAttribute: "subnetIds",
        Message_: "Subnets and security groups must belong to the default VPC when only one is provided."
      }
    status: "True"
    type: ACK.Recoverable
  - lastTransitionTime: "2022-11-22T01:12:40Z"
    message: Unable to determine if desired resource state matches latest observed
      state
    reason: |-
      BadRequestException: Subnets and security groups must belong to the default VPC when only one is provided.
      {
        RespMetadata: {
          StatusCode: 400,
          RequestID: "5ffd35ab-3d65-4776-ade4-cf126af0c997"
        },
        ErrorAttribute: "subnetIds",
        Message_: "Subnets and security groups must belong to the default VPC when only one is provided."
      }
    status: Unknown
    type: ACK.ResourceSynced

In Cloud Trail you see the go sdk sending an UpdateBroker call after the Broker is Running and includes in the parameters the security group, which is already set to that value.

"eventTime": "2022-11-22T00:37:42Z",
"eventSource": "amazonmq.amazonaws.com",
"eventName": "UpdateBroker",
"awsRegion": "us-east-1",
"sourceIPAddress": "18.235.50.27",
"userAgent": "aws-controllers-k8s/-v0.0.23 (GitCommit/740eedcdf35da5376ee0df354447476ca02d90f1; BuildDate/2022-11-15T17:22; CRDKind/; CRDVersion/) aws-sdk-go/1.44.93 (go1.17.13; linux; amd64)",
"errorCode": "BadRequestException",
"requestParameters": {
    "engineVersion": "3.10.10",
    "broker-id": "b-519d2e3f-c154-41c1-8a61-ffcb94c6b9f9",
    "securityGroups": [
        "sg-0db2eb2c65e795d4b"
    ],
    "autoMinorVersionUpgrade": false,
    "hostInstanceType": "mq.t3.micro"
},
"responseElements": {
    "message": "Changing security groups is not supported for RabbitMQ brokers.",
    "errorAttribute": "securityGroups"
},
"requestID": "2eabed1e-a66d-4613-8192-fedca30002d4",
"eventID": "1f396ee4-df9c-45ce-9079-e22878793a01",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,

In the logs

2022-11-22T01:20:45.606Z        INFO    ackrt   desired resource state has changed      {"account": "123456789", "role": "", "region": "us-east-1", "kind": "Broker", "namespace": "default", "name": "mq-eks-workshop", "is_adopted": false, "generation": 1, "diff": [{"Path":{"Parts":["Spec","AuthenticationStrategy"]},"A":null,"B":"simple"},{"Path":{"Parts":["Spec","EncryptionOptions"]},"A":null,"B":{"useAWSOwnedKey":true}},{"Path":{"Parts":["Spec","Logs"]},"A":null,"B":{"general":false}},{"Path":{"Parts":["Spec","MaintenanceWindowStartTime"]},"A":null,"B":{"dayOfWeek":"FRIDAY","timeOfDay":"22:00","timeZone":"UTC"}},{"Path":{"Parts":["Spec","StorageType"]},"A":null,"B":"ebs"}]}
2022-11-22T01:20:45.838Z        ERROR   controller.broker       Reconciler error        {"reconciler group": "mq.services.k8s.aws", "reconciler kind": "Broker", "name": "mq-eks-workshop", "namespace": "default", "error": "BadRequestException: Changing security groups is not supported for RabbitMQ brokers.\n{\n  RespMetadata: {\n    StatusCode: 400,\n    RequestID: \"a1e5ae66-86be-4624-8dc9-8dc631e6037f\"\n  },\n  ErrorAttribute: \"securityGroups\",\n  Message_: \"Changing security groups is not supported for RabbitMQ brokers.\"\n}"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227

I think having the Broker condition ACK.ResourceSynced not True affects the fieldexporters . 

status: Unknown
type: ACK.ResourceSynced

It looks to fix this we need to find a way to avoid to include security group on this API call, or avoid the API call since the values are the same that already set on the Broker.

a-hilaly commented 1 year ago

Looks like we have multiple issues in here:

csantanapr commented 1 year ago

@A-Hilaly any updates on this issue, I want to be able to show users how to use ACK with rabbitmq and currently is not working

ack-bot commented 1 year ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle stale

ack-bot commented 1 year ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle rotten

ack-bot commented 10 months ago

Issues go stale after 180d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 60d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle stale

gecube commented 6 months ago

/remove-lifecycle stale

ack-bot commented 1 week ago

Issues go stale after 180d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 60d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle stale

gecube commented 1 week ago

/remove-lifecycle stale