Cognito update attribute failure when running amplify push

pr0g commented 3 months ago

How did you install the Amplify CLI?

yarn

If applicable, what version of Node.js are you using?

v18.19.1

Amplify CLI Version

12.10.1

What operating system are you using?

macOS

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No

Describe the bug

When attempting to run amplify push, if there has been a failure somewhere (e.g. There was a GraphQL error for example), after correcting this, running it again will cause this failure:

🛑 The following resources failed to deploy:
Resource Name: UserPoolClientWeb (AWS::Cognito::UserPoolClient)
Event Type: update
Reason: Resource handler returned message: "Invalid write attributes specified while updating a client (Service: CognitoIdentityProvider, Status Code: 400, Request ID: <id>)" (RequestToken: <token>, HandlerErrorCode: InvalidRequest)
URL: <url>

Resource Name: UserPoolClient (AWS::Cognito::UserPoolClient)
Event Type: update
Reason: Resource handler returned message: "Invalid write attributes specified while updating a client (Service: CognitoIdentityProvider, Status Code: 400, Request ID: <id>)" (RequestToken: <token>, HandlerErrorCode: InvalidRequest)
URL: <url>

🛑 Resource is not in the state stackUpdateComplete
Name: UserPoolClientWeb (AWS::Cognito::UserPoolClient), Event Type: update, Reason: Resource handler returned message: "Invalid write attributes specified while updating a client (Service: CognitoIdentityProvider, Status Code: 400, Request ID: <id>)" (RequestToken: <token>, HandlerErrorCode: InvalidRequest), IsCustomResource: false

Name: UserPoolClient (AWS::Cognito::UserPoolClient), Event Type: update, Reason: Resource handler returned message: "Invalid write attributes specified while updating a client (Service: CognitoIdentityProvider, Status Code: 400, Request ID: <id>)" (RequestToken: <token>, HandlerErrorCode: InvalidRequest), IsCustomResource: false

If I then make a whitespace change to amplify/backend/auth/<app>/override.ts, and run amplify push again, things work as expected.

I have these settings in amplify/backend/auth/<app>/override.ts

export function override(resources: AmplifyAuthCognitoStackTemplate) {
  resources.userPool.schema = [
    {
      name: "email",
      required: true,
      mutable: true,
    },
    {
      name: "name",
      required: true,
      mutable: true,
    },
    {
      name: "company",
      required: false,
      mutable: true,
      attributeDataType: "String",
    },
    {
      name: "jobrole",
      required: false,
      mutable: true,
      attributeDataType: "String",
    },
    {
      name: "agreetcs",
      required: false,
      mutable: true,
      attributeDataType: "String",
    },
    {
      name: "marketing",
      required: false,
      mutable: true,
      attributeDataType: "String",
    },
    {
      name: "level",
      required: false,
      mutable: true,
      attributeDataType: "String",
    },
    {
      name: "marketsource",
      required: false,
      mutable: true,
      attributeDataType: "String",
    },
    {
      name: "created",
      required: false,
      mutable: true,
      attributeDataType: "String",
    },
  ];

  // set readable attributes
  resources.userPoolClient.readAttributes = [
    "address",
    "birthdate",
    "custom:agreetcs",
    "custom:company",
    "custom:created",
    "custom:jobrole",
    "custom:level",
    "custom:marketing",
    "custom:marketsource",
    "email",
    "email_verified",
    "family_name",
    "gender",
    "given_name",
    "locale",
    "middle_name",
    "name",
    "nickname",
    "phone_number",
    "phone_number_verified",
    "picture",
    "preferred_username",
    "profile",
    "updated_at",
    "website",
    "zoneinfo"
  ];

  resources.userPoolClientWeb.readAttributes =
    resources.userPoolClient.readAttributes;

  // set writable attributes
  resources.userPoolClient.writeAttributes = [
    "address",
    "birthdate",
    "custom:agreetcs",
    "custom:company",
    "custom:created",
    "custom:jobrole",
    "custom:level",
    "custom:marketing",
    "custom:marketsource",
    "email",
    "family_name",
    "gender",
    "given_name",
    "locale",
    "middle_name",
    "name",
    "nickname",
    "phone_number",
    "picture",
    "preferred_username",
    "profile",
    "updated_at",
    "website",
    "zoneinfo"
  ];

  resources.userPoolClientWeb.writeAttributes =
    resources.userPoolClient.writeAttributes;

  resources.userPool.emailConfiguration = {
    emailSendingAccount: "DEVELOPER",
    from: "<app-email>",
    sourceArn: "<arn>",
  };

I understand this is to do with Cognito thinking there's something off with the write attributes, but if I simply refresh override.ts with no functional changes, the deployment will succeed. It seems Amplify somehow gets into a bad state, and this can be fixed by making a simple change to override.ts

Expected behavior

amplify push works reliably without needing to modify amplify/backend/auth/<app>/override.ts after a failed deployment (after the original issue is resolved)

Reproduction steps

Run amplify push with auth settings specified in override.ts
Make a GraphQL change that is invalid (say a @connection that isn't correct potentially)
Run amplify push
See failure with stack deployment
Revert GraphQL change
Run amplify push again
See Cognito failure mentioned above

Project Identifier

6ea7243cd01dc9d083f59ce9c38bb874

Log output

``` 🛑 The following resources failed to deploy: Resource Name: UserPoolClientWeb (AWS::Cognito::UserPoolClient) Event Type: update Reason: Resource handler returned message: "Invalid write attributes specified while updating a client (Service: CognitoIdentityProvider, Status Code: 400, Request ID: )" (RequestToken: , HandlerErrorCode: InvalidRequest) URL: Resource Name: UserPoolClient (AWS::Cognito::UserPoolClient) Event Type: update Reason: Resource handler returned message: "Invalid write attributes specified while updating a client (Service: CognitoIdentityProvider, Status Code: 400, Request ID: )" (RequestToken: , HandlerErrorCode: InvalidRequest) URL: 🛑 Resource is not in the state stackUpdateComplete Name: UserPoolClientWeb (AWS::Cognito::UserPoolClient), Event Type: update, Reason: Resource handler returned message: "Invalid write attributes specified while updating a client (Service: CognitoIdentityProvider, Status Code: 400, Request ID: )" (RequestToken: , HandlerErrorCode: InvalidRequest), IsCustomResource: false Name: UserPoolClient (AWS::Cognito::UserPoolClient), Event Type: update, Reason: Resource handler returned message: "Invalid write attributes specified while updating a client (Service: CognitoIdentityProvider, Status Code: 400, Request ID: )" (RequestToken: , HandlerErrorCode: InvalidRequest), IsCustomResource: false ```

Additional information

N/A

Before submitting, please confirm:

[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
[X] I have removed any sensitive information from my code snippets and submission.

ykethan commented 3 months ago

Hey @pr0g, could you zip the amplify folder and email at amplify-cli@amazon.com? From the project identifier did notice the project cli.json is currently missing some feature-flags and the @connection indicates the application is utilizing graphQL transformer v1. Could add the following to cli.json

{
  "features": {
    "graphqltransformer": {
      "addmissingownerfields": true,
      "improvepluralization": false,
      "validatetypenamereservedwords": true,
      "useexperimentalpipelinedtransformer": true,
      "enableiterativegsiupdates": true,
      "secondarykeyasgsi": true,
      "skipoverridemutationinputtypes": true,
      "transformerversion": 2,
      "suppressschemamigrationprompt": true,
      "securityenhancementnotification": false,
      "showfieldauthnotification": false,
      "usesubusernamefordefaultidentityclaim": true,
      "usefieldnameforprimarykeyconnectionfield": false,
      "enableautoindexquerynames": true,
      "respectprimarykeyattributesonconnectionfield": true,
      "shoulddeepmergedirectiveconfigdefaults": false,
      "populateownerfieldforstaticgroupauth": true
    },
    "frontend-ios": {
      "enablexcodeintegration": true
    },
    "auth": {
      "enablecaseinsensitivity": true,
      "useinclusiveterminology": true,
      "breakcirculardependency": true,
      "forcealiasattributes": false,
      "useenabledmfas": true
    },
    "codegen": {
      "useappsyncmodelgenplugin": true,
      "usedocsgeneratorplugin": true,
      "usetypesgeneratorplugin": true,
      "cleangeneratedmodelsdirectory": true,
      "retaincasestyle": true,
      "addtimestampfields": true,
      "handlelistnullabilitytransparently": true,
      "emitauthprovider": true,
      "generateindexrules": true,
      "enabledartnullsafety": true,
      "generatemodelsforlazyloadandcustomselectionset": false
    },
    "appsync": {
      "generategraphqlpermissions": true
    },
    "latestregionsupport": {
      "pinpoint": 1,
      "translate": 1,
      "transcribe": 1,
      "rekognition": 1,
      "textract": 1,
      "comprehend": 1
    },
    "project": {
      "overrides": true
    }
  },
  "debug": {}
}

pr0g commented 3 months ago

Hi @ykethan,

Thanks very much for following-up. I don't have the issue reproducing at the moment so can't easily zip up our amplify folder. Should we hit this again I'll make sure to do that and add a note here for context.

I'm afraid we are still currently using GraphQL Transformer v1, we're hoping to upgrade in the coming months.

I can share our cli.json as it currently looks if that would help? Right now I don't think we can easily switch to transformer v2 without making a lot of changes to use the new API (@hasMany etc... instead of @connection in GraphQL for example).

Thanks very much for your time,

I'll do my best to reply as soon as this issue happens again

ykethan commented 3 months ago

@pr0g The transformer version is expected to not cause an issue with the auth override. do let us know if run into this.

pr0g commented 3 months ago

Thanks for following up @ykethan, that's good to know.

This may be tangentially related but I was very recently trying to add a custom resolver to provide the ability to append to an array in a DynamoDB record (see the SO answer).

amplify push worked once, but I think the auth rules I set were potentially wrong, but when I then try to update (make a change to GraphQL and do amplify push) it now is failing to redeploy and I get the error that the custom resource already exists (with the ARN of the resolver in AppSync).

The steps I followed are very similar to this doc page - https://docs.amplify.aws/javascript/tools/cli-legacy/overwrite-customize-resolvers/ where I add a new resource to CustomResource.json along with the GraphQL changes and vtl files.

I can send a zip of the amplify folder for that if that would help.

I haven't tried with the changes you suggested to cli.json, it's possible that might also make a difference.

If you have any suggestions I'd really appreciate it.

Thanks! 🙂

pr0g commented 3 months ago

This is the error I see for reference (<name-of-custom-resource> is the name used in stacks/CustomResources.json)

🛑 The following resources failed to deploy:
Resource Name: <name-of-custom-resource> (AWS::AppSync::Resolver)
Event Type: create
Reason: Resource handler returned message: "Resource already exists: arn:aws:appsync:eu-west-2:<id>:apis/<id>/types/Mutation/resolvers/<resolver-name>" (RequestToken: <id>, HandlerErrorCode: AlreadyExists)

🛑 Resource is not in the state stackUpdateComplete
Name: <name-of-custom-resource> (AWS::AppSync::Resolver), Event Type: create, Reason: Resource handler returned message: "Resource already exists: arn:aws:appsync:eu-west-2:<id>:apis/<id>/types/Mutation/resolvers/<resolver-name>" (RequestToken: <id>, HandlerErrorCode: AlreadyExists), IsCustomResource: false

Resource looks like this:

"<ResolverName>": {
      "Type": "AWS::AppSync::Resolver",
      "Properties": {
        "ApiId": {
          "Ref": "AppSyncApiId"
        },
        "DataSourceName": "SourceFileTable",
        "TypeName": "Mutation",
        "FieldName": "<function-name>", # same name used in GraphQL schema
        "RequestMappingTemplateS3Location": {
          "Fn::Sub": [
            "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/Mutation.<function-name>.req.vtl",
            {
              "S3DeploymentBucket": {
                "Ref": "S3DeploymentBucket"
              },
              "S3DeploymentRootKey": {
                "Ref": "S3DeploymentRootKey"
              }
            }
          ]
        },
        "ResponseMappingTemplateS3Location": {
          "Fn::Sub": [
            "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/Mutation.<function-name>.res.vtl",
            {
              "S3DeploymentBucket": {
                "Ref": "S3DeploymentBucket"
              },
              "S3DeploymentRootKey": {
                "Ref": "S3DeploymentRootKey"
              }
            }
          ]
        }
      }
    }

Thanks!

ykethan commented 3 months ago

@pr0g from the error message the issue appears to be occurring on a GraphQL API resource. I'm going to transfer this over to our API repository for better assistance.

pr0g commented 3 months ago

Okay sounds like a plan, thanks @ykethan!

pr0g commented 2 months ago

Any update from the aws-amplify/amplify-cli team?

Any info about the original problem (related to the switch from cli-inputs.json to overrides.ts for Cognito user pool attributes).

I'm seeing issues with these two properties in particular:

    {
      name: "level",
      required: false,
      mutable: true,
      attributeDataType: "String",
    },
    {
      name: "marketsource",
      required: false,
      mutable: true,
      attributeDataType: "String",
    },

Which I moved from cli-inputs.json to overrides.ts and in one environment, I can't do an amplify push anymore as the Cognito User Pool complains that these properties already exist. Is there a way to work around this (temporarily I've just commented these out but it's a pain when merging this change back and forward between two different environments).

Thanks for any help/ideas!

AnilMaktala commented 1 month ago

This issue seems to be related to #2458

pr0g commented 1 month ago

@AnilMaktala These two problems are unrelated.

What I've found is in my new environment, I have to keep these two attributes in overrides.ts uncommented, but in the old environment (where these attributes were added in cli-inputs.json) I have to leave them commented out 😖

This is making merging branches between environments a bit of a pain, but it's a reasonable workaround for now. If there's a way to unify both approaches I'd love to know!

AnilMaktala commented 1 month ago

Hey @pr0g, Thanks for confirming. "Regarding the existing property issue, have you tried running amplify pull for this environment in a different new root folder and then executing amplify push?"

pr0g commented 1 month ago

No problem at all @AnilMaktala

I can't say for certain I've done exactly that, but I've found that if I work out of my new environment, I have to have the lines below (^1) added, but in the old environment, they need to be commented out (see this post above).

This issue is very much related this this issue https://github.com/aws-amplify/amplify-cli/issues/13642 that happened when I first tried to clone a new environment a little while back. It might just be something we have to live with, as the old environment was created in a legacy way potentially (using cli-inputs.json instead of override.ts).

[1]

    {
      name: "level",
      required: false,
      mutable: true,
      attributeDataType: "String",
    },
    {
      name: "marketsource",
      required: false,
      mutable: true,
      attributeDataType: "String",
    },

aws-amplify / amplify-category-api