ArroyoSystems / arroyo

Distributed stream processing engine in Rust
https://arroyo.dev
Apache License 2.0
3.67k stars 206 forks source link

Allow to configure worker's IRSA through worker's ServiceAccount annotation #645

Closed rparme closed 2 months ago

rparme commented 3 months ago

When it comes to connecting to an S3 bucket, the documentation and the blog suggest to either use aws_access_key_id / aws_secret_access_key or to grant permission to the Kubernetes node IAM role.

I believe this misses a more granular permission mechanism and I suggest to add to possibility to define the annotations of the ServiceAccount the workers run with so that an IRSA can be linked, as already done for the controller's ServiceAccount.

An even more granular possibility would be to configure the used IRSA in the backend configuration: either configure storage.aws_secret_access_key + storage.aws_access_key_id or (new) storage.iam_role_for_service_account (this means we know the namespace and the name of the ServiceAccount as it is required to create the IRSA).