Creating SNS subscription with SQS Protocol

rj425 commented 1 year ago

I have a SNS topic and SQS queue. I would like to have my SQS queue subscribed to SNS topic. I have looked at the list of CRDs that SQS and SNS controller offer. But i am unable to find the right CRD that can help me do so.

What have i tried so far:

I created a subscription object from the AWS console to understand the configuration changes that takes place in the policy. I tried to make these exact changes to SQS policy using SQS controller. The policy gets changed successfully but there is no SNS subscription.
I have looked at the PlatformApplication and PlatformEndpoint CRDs in SNS controller, but i fail to understand if these two CRDs are the solution. If that is the case, can you point me to the documentation that explains it well.

Thanks you.

a-hilaly commented 1 year ago

Currently we are not able to regenerate the sns-controller/sqs-controller and any new resource because of the problems mentioned in https://github.com/aws-controllers-k8s/community/issues/1292

jessebye commented 1 year ago

Hi @A-Hilaly, issue #1292 is now closed. Would you be able to regenerate the SNS controller resources at this point? We need to create SNS subscriptions as well, and are blocked until that is added.

a-hilaly commented 1 year ago

Hi @jessebye the sns-controller have been regenerated and a new release v0.0.4 was released? currently it doesn't support Subscriptions but we can definitely open an issue and give it a shot.

a-hilaly commented 1 year ago

@jessebye I just checked the API and i don't see a CreateSubscription API Call, however there is a Subscribe call that is related to SNS Endpoint and SNS Topics, is this the operation you'd like to see supported?

jessebye commented 1 year ago

Hi @A-Hilaly , thanks for the update! Yes, Subscribe would be the operation we want to support. Also probably ConfirmSubscription so we can confirm the subscription after creating it. Thank you 😀

jaypipes commented 1 year ago

Hi @A-Hilaly , thanks for the update! Yes, Subscribe would be the operation we want to support. Also probably ConfirmSubscription so we can confirm the subscription after creating it. Thank you grinning

@jessebye @A-Hilaly I suppose we could try to create a Subscription resource and have Subscribe be the Create call and Unsubscribe be the Delete call. We would use the GetSubscriptionAttributes and SetSubscriptionAttributes calls for the Update code path.

This would come with a caveat, though:

We would only support creating Subscription resources to endpoints in the same AWS account. The reason for this is because the requirement for the endpoint owner to call ConfirmSubscription when receiving a Subscribe call from an account other than the endpoint's owner is antithetical to the way that Kubernetes controllers work to reconcile the state of an owned resource. In the case of Subscribing to a different account's Endpoint, the subscription isn't actually owned by the calling account.

@jessebye would you be cool with the above behaviour?

jessebye commented 1 year ago

@jaypipes sounds reasonable to me! Thanks!

jessebye commented 1 year ago

@jaypipes @A-Hilaly Is there anything I can do to help move this along? This is blocking our team's progress on some initiatives that use SNS. I'd be glad to contribute if that would be helpful!

jaypipes commented 1 year ago

@jaypipes @A-Hilaly Is there anything I can do to help move this along? This is blocking our team's progress on some initiatives that use SNS. I'd be glad to contribute if that would be helpful!

Working on this right now, @jessebye! :) Just got done adding FIFOTopic and other new attributes to the controller. Adding support for Subscription resource now...

matheushent commented 1 year ago

1292

Hi Jay, it's been a little bit more than a month and I know you're pretty much busy implementing all the stuff we ask, but, can you give an update on the implementation status?

jaypipes commented 1 year ago

@matheushent I've made great progress on it. Should have something pushed by end of today.

matheushent commented 1 year ago

@jaypipes awesome mate, do you have the PR or branch so I can follow?

jaypipes commented 1 year ago

@matheushent yup! here it is: https://github.com/jaypipes/ack-sns-controller/tree/subscription

I'm working on e2e tests and debugging failures :)

jaypipes commented 1 year ago

@matheushent @jessebye @rj425 OK all, I have support for SNS Subscriptions in a PR here:

https://github.com/aws-controllers-k8s/sns-controller/pull/26

Hoping to get that merged today and work on support for changing Subscription Attributes and setting tags on Subscription resources.

jaypipes commented 1 year ago

@matheushent @jessebye @rj425 Release v0.0.11 contains support for Subscription resources. Please do check it out.

logand22 commented 1 year ago

Would something like EndpointRef be possible with the Subscription resource? Would maybe be a QoL improvement when working with Topic resource and SQS Queue resource.

jaypipes commented 1 year ago

Would something like EndpointRef be possible with the Subscription resource? Would maybe be a QoL improvement when working with Topic resource and SQS Queue resource.

@logand22 which specific field on Subscription resources do you want to add resource reference ability?

logand22 commented 1 year ago

@jaypipes My particular use case would be to allow you to use resource reference with different endpoints. For instance, creating a subscription between SNS + SQS when running both the sns + sqs controllers. I can do it today using the ARN, but it would be a quality of life improvement to use the SQS Queue resource name.

ack-bot commented 8 months ago

Issues go stale after 180d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 60d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle stale

ack-bot commented 6 months ago

Stale issues rot after 60d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 60d of inactivity. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle rotten

ack-bot commented 1 week ago

Issues go stale after 180d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 60d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle stale

aws-controllers-k8s / community

Creating SNS subscription with SQS Protocol #1491

1292