search

aws-controllers-k8s / community

AWS Controllers for Kubernetes (ACK) is a project enabling you to manage AWS services from Kubernetes
https://aws-controllers-k8s.github.io/community/
Apache License 2.0
2.4k stars 253 forks source link

EC2 controller - no way of managing default security group in VPC #1896

Open gecube opened 1 year ago

gecube commented 1 year ago

Good day!

I am playing around with EC2 controller and found that basic creation of VPC with the manifest like 

apiVersion: ec2.services.k8s.aws/v1alpha1
kind: VPC
metadata:
  name: dev
  namespace: infra-dev
spec:
  cidrBlocks:
    - 10.10.0.0/16
  enableDNSSupport: true
  enableDNSHostnames: true
  tags:
    - key: Name
      value: dev

leads to creation default security group.

Screenshot 2023-09-14 at 9 38 09 Screenshot 2023-09-14 at 9 38 22

Unfortunately, this security group has allow all rules for inbound and outbound connections and fails security check:

Screenshot 2023-09-14 at 9 37 30

If I create additional security groups, they are created well.

I'd like to have nice and clean way of managing this "default" security group in terms of EC2 controller objects. Probably - the adoption pattern won't be very good here, as an operator of ACK I want to create all relevant objects and configure them in one go. I am kindly asking to give suggestion and options how to achieve the desired state.

gecube commented 1 year ago

Some more clues.

I can't create a default security group from code directly. It won't work as controller throws error:

2023-09-14T07:50:02.695Z    ERROR   Reconciler error    {"controller": "securitygroup", "controllerGroup": "ec2.services.k8s.aws", "controllerKind": "SecurityGroup", "SecurityGroup": {"name":"default","namespace":"infra-dev"}, "namespace": "infra-dev", "name": "default", "reconcileID": "138bd869-c9c9-457e-829f-a73fe0698afc", "error": "InvalidParameterValue: Cannot use reserved security group name: default\n\tstatus code: 400, request id: d85c0a9a-ff92-4b28-8acd-610732d1a698"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235

Also there is no direct way to adopt this sg. If I try to apply manifest like:

apiVersion: services.k8s.aws/v1alpha1
kind: AdoptedResource
metadata:
  name: adopt-my-existing-bucket
  namespace: infra-dev
spec:
  aws:
    nameOrID: default
  kubernetes:
    group: ec2.services.k8s.aws
    kind: SecurityGroup
    metadata:
      name: default
      namespace: infra-dev

The EC2 controllers gives me in status:

  conditions:
    - message: "InvalidGroupId.Malformed: Invalid id: \"default\" (expecting \"sg-...\")\n\tstatus code: 400, request id: 56865034-285f-4356-ae9a-29efcd1ee9c2"
      status: 'False'
      type: ACK.Adopted

But I can't retrieve the Group ID directly from VPC description... Everything I know about VPC is written directly into the status field of VPC:

status:
  ackResourceMetadata:
    ownerAccountID: '178394743802'
    region: eu-west-2
  cidrBlockAssociationSet:
    - associationID: vpc-cidr-assoc-0f97c3c42baf28acf
      cidrBlock: 10.10.0.0/16
      cidrBlockState:
        state: associated
  conditions:
    - lastTransitionTime: '2023-09-14T01:42:47Z'
      message: Resource synced successfully
      reason: ''
      status: 'True'
      type: ACK.ResourceSynced
  dhcpOptionsID: dopt-b0cbf6d8
  isDefault: false
  ownerID: '178394743802'
  state: available
  vpcID: vpc-0017152cc2d43a69a

and there is no security group id, but it could be logical to add it there.

gecube commented 1 year ago

O.K. so I can retrieve the security group name from Amazon Console and substitute it in YAML:

apiVersion: services.k8s.aws/v1alpha1
kind: AdoptedResource
metadata:
  name: default-security-group
  namespace: infra-dev
spec:
  aws:
    nameOrID: sg-0e87e0dd9f6d43f31
  kubernetes:
    group: ec2.services.k8s.aws
    kind: SecurityGroup
    metadata:
      name: default
      namespace: infra-dev

and then I am getting the cryptic error message like:

status:
  conditions:
    - message: >-
        SecurityGroup.ec2.services.k8s.aws "default" is invalid: spec.name:
        Required value
      status: 'False'
      type: ACK.Adopted

and

2023-09-14T08:08:46.656Z    ERROR   Reconciler error    {"controller": "adoptedresource", "controllerGroup": "services.k8s.aws", "controllerKind": "AdoptedResource", "AdoptedResource": {"name":"default-security-group","namespace":"infra-dev"}, "namespace": "infra-dev", "name": "default-security-group", "reconcileID": "d860b0a9-2f5b-48fe-afd7-324ab67a8394", "error": "SecurityGroup.ec2.services.k8s.aws \"default\" is invalid: spec.name: Required value"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235

in EC2 controller logs

No idea what does it mean.

a-hilaly commented 1 year ago

@gecube Can you try settings metadata.name to something different than default?

Regarding the default security group created with the VPC, maybe we could consider deleting it right after a VPC creation. I wish there was a way to create a subnet-less-vpc in https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateVpc.html

gecube commented 1 year ago

@a-hilaly Hi! The same:

spec:
  aws:
    nameOrID: sg-0e87e0dd9f6d43f31
  kubernetes:
    group: ec2.services.k8s.aws
    kind: SecurityGroup
    metadata:
      name: default-2
      namespace: infra-dev
status:
  conditions:
    - message: >-
        SecurityGroup.ec2.services.k8s.aws "default-2" is invalid: spec.name:
        Required value
      status: 'False'
      type: ACK.Adopted

logs:

{"level":"error","ts":"2023-09-15T08:11:00.409Z","msg":"Reconciler error","controller":"adoptedresource","controllerGroup":"services.k8s.aws","controllerKind":"AdoptedResource","AdoptedResource":{"name":"default-security-group-2","namespace":"infra-dev"},"namespace":"infra-dev","name":"default-security-group-2","reconcileID":"846a56de-f57b-4f9f-89a7-840ced5b3dc1","error":"SecurityGroup.ec2.services.k8s.aws \"default-2\" is invalid: spec.name: Required value","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235"}
{"level":"info","ts":"2023-09-15T08:12:22.335Z","logger":"adoption.adopted-reconciler","msg":"starting adoption reconciliation","target_group":"ec2.services.k8s.aws","target_kind":"SecurityGroup","namespace":"infra-dev","name":"default-security-group-2","generation":1}
{"level":"error","ts":"2023-09-15T08:12:22.446Z","msg":"Reconciler error","controller":"adoptedresource","controllerGroup":"services.k8s.aws","controllerKind":"AdoptedResource","AdoptedResource":{"name":"default-security-group-2","namespace":"infra-dev"},"namespace":"infra-dev","name":"default-security-group-2","reconcileID":"1c7c782a-d3c4-40c5-b518-083ec65cdb04","error":"SecurityGroup.ec2.services.k8s.aws \"default-2\" is invalid: spec.name: Required value","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235"}
mattzech commented 11 months ago

Any update on this? We are facing the same issue. Thanks!

ack-bot commented 5 months ago

Issues go stale after 180d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 60d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle stale

gecube commented 5 months ago

/remove-lifecycle stale