sicoyle commented 1 week ago

Description

This PR:

Adds the IAM Roles Anywhere auth profile to Kafka
Moves the disjoint Assume Role fields to their own auth profile. This is only used by a few components, so the description is marked as such. However, it is an auth profile that we can add support for in other components, so I've moved it to the auth profile common AWS area to allow us to add this to additional components in the future.
Denoted that the originally disjoint Kafka AWS fields become deprecated in Dapr 1.17 as we mv everything to use the common AWS auth profiles that are built in. This meant really gross manual manipulations on the metadata bundle unfortunately...

Metadata bundle changes: NOTE: This includes all duplicate fields as we deprecate some in the future. Due to this, I've manually made things like accessKey and secretKey that are normally required, not required to not break existing users as we migrate to the common AWS auth fields without the aws prefix.

{
          "title": "AWS: Access Key ID and Secret Access Key",
          "description": "Authenticate using an Access Key ID and Secret Access Key included in the metadata",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            },
            {
              "name": "awsAccessKey",
              "description": "Deprecated as of Dapr 1.17. Use 'accessKey' instead.\nIf both fields are set, then 'accessKey' value will be used.\nAWS access key associated with an IAM account.",
              "type": "string",
              "example": "\"AKIAIOSFODNN7EXAMPLE\""
            },
            {
              "name": "awsSecretKey",
              "description": "Deprecated as of Dapr 1.17. Use 'secretKey' instead.\nIf both fields are set, then 'secretKey' value will be used.\nThe secret key associated with the access key.",
              "sensitive": true,
              "type": "string",
              "example": "\"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\""
            },
            {
              "name": "awsSessionToken",
              "description": "Deprecated as of Dapr 1.17. Use 'sessionToken' instead.\nIf both fields are set, then 'sessionToken' value will be used.\nAWS session token to use. A session token is only required if you are using temporary security credentials.",
              "sensitive": true,
              "type": "string",
              "example": "\"TOKEN\""
            },
            {
              "name": "awsIamRoleArn",
              "description": "Deprecated as of Dapr 1.17. Use 'assumeRoleArn' instead.\nIf both fields are set, then 'assumeRoleArn' value will be used.\nIAM role that has access to MSK. This is another option to authenticate with MSK aside from the AWS Credentials.",
              "type": "string",
              "example": "\"arn:aws:iam::123456789:role/mskRole\""
            },
            {
              "name": "awsStsSessionName",
              "description": "Deprecated as of Dapr 1.17. Use 'sessionName' instead.\nIf both fields are set, then 'sessionName' value will be used.\nRepresents the session name for assuming a role.",
              "type": "string",
              "default": "\"MSKSASLDefaultSession\"",
              "example": "\"MyAppSession\""
            },
            {
              "name": "awsRegion",
              "description": "The AWS Region where the AWS resource is deployed to.",
              "required": true,
              "type": "string",
              "example": "\"us-east-1\""
            },
            {
              "name": "accessKey",
              "description": "AWS access key associated with an IAM account",
              "required": true,
              "sensitive": true,
              "example": "\"AKIAIOSFODNN7EXAMPLE\""
            },
            {
              "name": "secretKey",
              "description": "The secret key associated with the access key",
              "required": true,
              "sensitive": true,
              "example": "\"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\""
            },
            {
              "name": "sessionToken",
              "description": "AWS session token to use. A session token is only required if you are using\ntemporary security credentials.",
              "sensitive": true,
              "example": ""
            }
          ]
        },

Note: this is a new auth profile I've made by moving some of the fields here, so this shows some fields that are becoming deprecated:

        {
          "title": "AWS: Assume specific IAM Role",
          "description": "Assume a specific IAM role. Note: This is only supported on Kafka and PostgreSQL components.",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            },
            {
              "name": "awsIamRoleArn",
              "description": "Deprecated as of Dapr 1.17. Use 'assumeRoleArn' instead.\nIf both fields are set, then 'assumeRoleArn' value will be used.\nIAM role that has access to MSK. This is another option to authenticate with MSK aside from the AWS Credentials.",
              "type": "string",
              "example": "\"arn:aws:iam::123456789:role/mskRole\""
            },
            {
              "name": "awsStsSessionName",
              "description": "Deprecated as of Dapr 1.17. Use 'sessionName' instead.\nIf both fields are set, then 'sessionName' value will be used.\nRepresents the session name for assuming a role.",
              "type": "string",
              "default": "\"MSKSASLDefaultSession\"",
              "example": "\"MyAppSession\""
            },
            {
              "name": "sessionToken",
              "description": "AWS session token to use. A session token is only required if you are using\ntemporary security credentials.",
              "sensitive": true,
              "type": "string",
              "example": "\"TOKEN\""
            },
            {
              "name": "assumeRoleArn",
              "description": "IAM role that has access to AWS resource.\nThis is another option to authenticate with MSK and RDS Aurora aside from the AWS Credentials.",
              "type": "string",
              "example": "\"arn:aws:iam::123456789:role/mskRole\""
            },
            {
              "name": "sessionName",
              "description": "The session name for assuming a role.",
              "type": "string",
              "default": "\"MSKSASLDefaultSession\"",
              "example": "\"MyAppSession\""
            }
          ]
        },

Note: This remains as is with the manual manipulation to remove the deprecated fields that are unused for this profile:

        {
          "title": "AWS: Credentials from Environment Variables",
          "description": "Use AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the environment",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            }
          ]
        },

Note: new profile added that I manually remove all the fields that we are deprecating bc they are not needed for this auth profile

        {
          "title": "AWS: IAM Roles Anywhere",
          "description": "Use X.509 certificates to establish trust between AWS and your AWS account and the Dapr cluster using AWS IAM Roles Anywhere.",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            },
            {
              "name": "trustAnchorArn",
              "description": "ARN of the AWS Trust Anchor in the AWS account granting trust to the Dapr Certificate Authority.",
              "required": true,
              "example": "arn:aws:rolesanywhere:us-west-1:012345678910:trust-anchor/01234568-0123-0123-0123-012345678901"
            },
            {
              "name": "trustProfileArn",
              "description": "ARN of the AWS IAM Profile in the trusting AWS account.",
              "required": true,
              "example": "arn:aws:rolesanywhere:us-west-1:012345678910:profile/01234568-0123-0123-0123-012345678901"
            },
            {
              "name": "assumeRoleArn",
              "description": "ARN of the AWS IAM role to assume in the trusting AWS account.",
              "required": true,
              "example": "arn:aws:iam:012345678910:role/exampleIAMRoleName"
            }
          ]
        }

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #[issue number]

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

[ ] Code compiles correctly
[ ] Created/updated tests
[ ] Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

sicoyle commented 1 week ago

/ok-to-test

sicoyle commented 1 week ago

unit tests are failing due to wasm issues that are unrelated

dapr-bot commented 1 week ago

Complete Build Matrix

The build status is currently not updated here. Please visit the action run below directly.

🔗 Link to Action run

Commit ref: a1f1c9a91654d68b53b1350b9d0c979b8c6005ae

dapr-bot commented 1 week ago

Components conformance test

🔗 Link to Action run

Commit ref: a1f1c9a91654d68b53b1350b9d0c979b8c6005ae

❌ Some conformance tests failed

These tests failed:

bindings.azure.eventgrid

dapr-bot commented 1 week ago

Components certification test

🔗 Link to Action run

Commit ref: a1f1c9a91654d68b53b1350b9d0c979b8c6005ae

❌ Some certification tests failed

These tests failed:

pubsub.aws.snssqs
pubsub.gcp.pubsub
state.azure.cosmosdb
state.sqlserver

famarting commented 6 days ago

overall lgtm

sicoyle commented 6 days ago

/ok-to-test

dapr-bot commented 6 days ago

Components certification test

🔗 Link to Action run

Commit ref: 16834edb1504a7e8f3d7bc9daf143e1f1ea7deb7

❌ Some certification tests failed

These tests failed:

pubsub.aws.snssqs
pubsub.gcp.pubsub
state.azure.cosmosdb
state.sqlserver

dapr-bot commented 6 days ago

Complete Build Matrix

The build status is currently not updated here. Please visit the action run below directly.

🔗 Link to Action run

Commit ref: 16834edb1504a7e8f3d7bc9daf143e1f1ea7deb7

dapr-bot commented 6 days ago

Components conformance test

🔗 Link to Action run

Commit ref: 16834edb1504a7e8f3d7bc9daf143e1f1ea7deb7

❌ Some conformance tests failed

These tests failed:

bindings.azure.eventgrid
state.azure.cosmosdb
state.sqlserver

sicoyle commented 6 days ago

final metadata updates

{
          "title": "AWS: Access Key ID and Secret Access Key",
          "description": "Authenticate using an Access Key ID and Secret Access Key included in the metadata",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            },
            {
              "name": "awsRegion",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'region' instead.\nThe AWS Region where the AWS Relational Database Service is deployed to.",
              "type": "string",
              "example": "\"us-east-1\""
            },
            {
              "name": "awsAccessKey",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'accessKey' instead.\nIf both fields are set, then 'accessKey' value will be used.\nAWS access key associated with an IAM account.",
              "type": "string",
              "example": "\"AKIAIOSFODNN7EXAMPLE\""
            },
            {
              "name": "awsSecretKey",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'secretKey' instead.\nIf both fields are set, then 'secretKey' value will be used.\nThe secret key associated with the access key.",
              "sensitive": true,
              "type": "string",
              "example": "\"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\""
            },
            {
              "name": "awsSessionToken",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'sessionToken' instead.\nIf both fields are set, then 'sessionToken' value will be used.\nAWS session token to use. A session token is only required if you are using temporary security credentials.",
              "sensitive": true,
              "type": "string",
              "example": "\"TOKEN\""
            },
            {
              "name": "awsIamRoleArn",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'assumeRoleArn' instead.\nIf both fields are set, then 'assumeRoleArn' value will be used.\nIAM role that has access to MSK. This is another option to authenticate with MSK aside from the AWS Credentials.",
              "type": "string",
              "example": "\"arn:aws:iam::123456789:role/mskRole\""
            },
            {
              "name": "awsStsSessionName",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'sessionName' instead.\nIf both fields are set, then 'sessionName' value will be used.\nRepresents the session name for assuming a role.",
              "type": "string",
              "default": "\"MSKSASLDefaultSession\"",
              "example": "\"MyAppSession\""
            },
            {
              "name": "region",
              "description": "The AWS Region where the AWS resource is deployed to.",
              "type": "string",
              "example": "\"us-east-1\""
            },
            {
              "name": "accessKey",
              "description": "AWS access key associated with an IAM account",
              "sensitive": true,
              "example": "\"AKIAIOSFODNN7EXAMPLE\""
            },
            {
              "name": "secretKey",
              "description": "The secret key associated with the access key",
              "sensitive": true,
              "example": "\"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\""
            },
            {
              "name": "sessionToken",
              "description": "AWS session token to use. A session token is only required if you are using\ntemporary security credentials.",
              "sensitive": true,
              "example": ""
            }
          ]
        },
        {
          "title": "AWS: Assume specific IAM Role",
          "description": "Assume a specific IAM role. Note: This is only supported on Kafka and PostgreSQL components.",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            },
            {
              "name": "awsRegion",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'region' instead.\nThe AWS Region where the AWS Relational Database Service is deployed to.",
              "type": "string",
              "example": "\"us-east-1\""
            },
            {
              "name": "awsIamRoleArn",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'assumeRoleArn' instead.\nIf both fields are set, then 'assumeRoleArn' value will be used.\nIAM role that has access to MSK. This is another option to authenticate with MSK aside from the AWS Credentials.",
              "type": "string",
              "example": "\"arn:aws:iam::123456789:role/mskRole\""
            },
            {
              "name": "awsStsSessionName",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'sessionName' instead.\nIf both fields are set, then 'sessionName' value will be used.\nRepresents the session name for assuming a role.",
              "type": "string",
              "default": "\"MSKSASLDefaultSession\"",
              "example": "\"MyAppSession\""
            },
            {
              "name": "region",
              "description": "The AWS Region where the AWS resource is deployed to.",
              "type": "string",
              "example": "\"us-east-1\""
            },
            {
              "name": "assumeRoleArn",
              "description": "IAM role that has access to AWS resource.\nThis is another option to authenticate with MSK and RDS Aurora aside from the AWS Credentials.",
              "type": "string",
              "example": "\"arn:aws:iam::123456789:role/mskRole\""
            },
            {
              "name": "sessionName",
              "description": "The session name for assuming a role.",
              "type": "string",
              "default": "\"MSKSASLDefaultSession\"",
              "example": "\"MyAppSession\""
            }
          ]
        },
        {
          "title": "AWS: Credentials from Environment Variables",
          "description": "Use AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the environment",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            }
          ]
        },
        {
          "title": "AWS: IAM Roles Anywhere",
          "description": "Use X.509 certificates to establish trust between AWS and your AWS account and the Dapr cluster using AWS IAM Roles Anywhere.",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            },
            {
              "name": "trustAnchorArn",
              "description": "ARN of the AWS Trust Anchor in the AWS account granting trust to the Dapr Certificate Authority.",
              "required": true,
              "example": "arn:aws:rolesanywhere:us-west-1:012345678910:trust-anchor/01234568-0123-0123-0123-012345678901"
            },
            {
              "name": "trustProfileArn",
              "description": "ARN of the AWS IAM Profile in the trusting AWS account.",
              "required": true,
              "example": "arn:aws:rolesanywhere:us-west-1:012345678910:profile/01234568-0123-0123-0123-012345678901"
            },
            {
              "name": "assumeRoleArn",
              "description": "ARN of the AWS IAM role to assume in the trusting AWS account.",
              "required": true,
              "example": "arn:aws:iam:012345678910:role/exampleIAMRoleName"
            }
          ]
        }

dapr / components-contrib

feat(kafka): iam roles anywhere + assume role auth profiles #3606

Description

Issue reference

Checklist

Complete Build Matrix

Components conformance test

❌ Some conformance tests failed

Components certification test

❌ Some certification tests failed

Components certification test

❌ Some certification tests failed

Complete Build Matrix

Components conformance test

❌ Some conformance tests failed