search

kubernetes-sigs / cluster-api-provider-aws

Kubernetes Cluster API Provider AWS provides consistent deployment and day 2 operations of "self-managed" and EKS Kubernetes clusters on AWS.
http://cluster-api-aws.sigs.k8s.io/
Apache License 2.0
634 stars 560 forks source link

Lifecycle Hooks for MachinePool/ASG #4020

Open sebltm opened 1 year ago

sebltm commented 1 year ago

/kind feature

Describe the solution you'd like ASGs have support for Lifecycle Hooks and it's a good idea to expose this in the MachinePool CRD. When I scale up and down my ASGs, I want to be able to execute some Lambda functions to create and delete other resources that my nodes need. Doing this through the lifecycle hook allows me to only allow the creation of the EC2 instance to go ahead if the Lambda function ran successfully.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

k8s-ci-robot commented 1 year ago

This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
dlipovetsky commented 1 year ago

@sebltm I think you mean the AWSMachinePool CRD.

Please describe how the hooks work with ASGs. Where do the hooks get defined in the ASG API? Do they have to be implemented using AWS Lambda? What changes to IAM policies would be required? Thanks!

/triage needs-information

Ankitasw commented 1 year ago

@dlipovetsky Here are some links regarding lifecycle hooks: https://docs.aws.amazon.com/autoscaling/ec2/userguide/lifecycle-hooks.html https://docs.aws.amazon.com/autoscaling/ec2/userguide/adding-lifecycle-hooks.html#adding-lifecycle-hooks-aws-cli

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

sebltm commented 1 year ago

Sorry I’d forgotten about this one! The hooks can target SNS, SQS or EventBridge - whether there’s also a Lambda or something else shouldn’t be part of the scope of this implementation. Rather should be led to the user to implement in my opinion. We should just add a lifecycleHook with the ARN of the resource they’ve decided to use:

lifecycleHook:
  sns:
  sqs:
  eventBridge:

(Only one of the options above should be mentioned) I’d be interested in working on it

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 year ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-aws/issues/4020#issuecomment-1642760619): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
sebltm commented 6 months ago

Hi, can we reopen this issue I'd like to take a look at it

sebltm commented 6 months ago

@dlipovetsky Lifecycle hooks pause the process of scaling in or scaling out instances with a notification system. Notifications are sent to SQS, SNS or EventBridge and a thirdparty system processes the notification, takes some actions and then allows or denies continuing the scale in/out actions.

They are part of the ASG definition in the API, so the user would need to define as a part of the AWSMachinePool or AWSManagedMachinePool which ARN is going to process the notificaiton (ARN of the SNS or SQS queue for example). The actual management process of the SNS/SQS/EventBridge should not be in scope for these, we can assume the user created the appropriate infrastructure through a thirdparty system. Therefore no changes to the IAM policies are required.

The lifecycle hooks need to be created after the ASG. I propose adding the AWSMachinePool and AWSManagedMachinePool CRDs a field:

lifecycleHooks:
- name: <name of the lifecycle hook; required>
  notificationTargetARN: <ARN of a target like an SQS or SNS queue otherwise the default is EventBridge; optional>
  roleARN: <role ARN to use for the hook; optional>
  lifecycleTransition: <the instance state to which to attach the hook; required>
  heartbeatTimeout: <the timeout in seconds associated to the hook after which the default action is used, default is 3600 seconds; optional>
  defaultResult: <action taken after timeout, either CONTINUE or ABANDON, default is ABANDON; optional>
  notificationMetadata: <some additional string data passed to the target; optional>

which would be reconciled after the ASG is created and deleted after the ASG is deleted.

sebltm commented 4 months ago

/reopen

k8s-ci-robot commented 4 months ago

@sebltm: Reopened this issue.

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-aws/issues/4020#issuecomment-2080445269): >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
sebltm commented 4 months ago

/remove-lifecycle rotten

sebltm commented 3 months ago

@dlipovetsky any more information needed on this one?

sebltm commented 3 months ago

@Ankitasw @dlipovetsky how can I help move this one forward please? :)

k8s-triage-robot commented 1 day ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale