raulb
commented
2 months ago 👋 Thank you @gedw99 for showing your interest in making this connector more flexible. I agree that having the connector able to subscribe to these notifications would make it more efficient. I can't make promises on how soon we'll get this done since we have other priorities in the works, but we'll keep this one on our radar for future implementation. If you're willing to open a pull request implementing this, we'd be happy to assist with anything needed there.
Thank you again!
Feature description
Hey
This plugin uses polling. Would be interesting if it could support Notifications.
AWS S3
https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html
Minio
https://min.io/docs/minio/linux/administration/monitoring/bucket-notifications.html
Google Cloud Storage
https://github.com/conduitio-labs/conduit-connector-google-cloudstorage
What are the things to think about if we did want to do this ?
On receiving the Notification we could then do what the poller did, and so the OpenCDC output would be the same. So no changes to any common code in the Conduit system.
Not all S3 systems support it perhaps, but we could do it on a case by case basis.
The Notification system might not be a CloudEvent.
Notification format conversion. Each Provider has a different way of describing the Notification, and so we would have to convert them. We are only interested in the File change Delta, and not things like Security aspects liked ACL's.