Join Table Issue after a failed build - Cannot update GSI's properties other than Provisioned Throughput and Contributor Insights Specification. You can create a new GSI with a different name.

Taylor-S commented 1 year ago

Before opening, please confirm:

[X] I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
[X] I have searched for duplicate or closed issues.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
[X] I have removed any sensitive information from my code snippets and submission.

How did you install the Amplify CLI?

No response

If applicable, what version of Node.js are you using?

No response

Amplify CLI Version

10.4.1

What operating system are you using?

Windows

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No I did not.

Amplify Categories

api

Amplify Commands

push

Describe the bug

I have CI/CD setup with my github account and pushed my changes to the dev environment. This threw an error in the build step of the deployment. Error stating: "This issue likely happens when the project has been pushed with a newer version of Amplify CLI, try updating to a newer version."

I then upgraded to the latest version being 10.4.1 at this current time and redeployed. (Also updated the Live Package Overrides in build settings to meet this) I was previously working with CLI version 9.2.1

It gets to the stage where it updates my dynamodb tables and resolvers and gets to a join table "ParentsStudents". I get the following message:

UPDATE_FAILED ParentsStudentsTable AWS::DynamoDB::Table Mon Nov 14 2022 03:30:21 GMT+0000 (Coordinated Universal Time) Resource handler returned message: "Cannot update GSI's properties other than Provisioned Throughput and Contributor Insights Specification. You can create a new GSI with a different name." (RequestToken: e68fba5c-87e9-4245-9271-97e71983c3f1, HandlerErrorCode: InvalidRequest)

The build ultimately finishes with the following error messages:

UPDATE_ROLLBACK_COMPLETE amplify-teacherapp-dev-134619 AWS::CloudFormation::Stack Mon Nov 14 2022 03:33:31 GMT+0000 (Coordinated Universal Time) 2022-11-14T03:33:56.478Z [INFO]: Following resources failed Resource Name: arn:aws:appsync:ap-southeast-2:198433734666:apis/vhcniphckraatpz7ywxjbmp6ta/types/Subscription/resolvers/onDeleteParentsStudents (AWS::AppSync::Resolver) Event Type: update Reason: Resource update cancelled URL: https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-2#/stacks/arn%3Aaws%3Acloudformation%3Aap-southeast-2%3A198433734666%3Astack%2Famplify-teacherapp-dev-134619-apiteacherapp-1DASTOQLVMX22-ParentsStudents-1597ZLBS0Z09S%2Fb5581010-1ed8-11ed-92c3-066fa68469a2/events Resource Name: arn:aws:appsync:ap-southeast-2:198433734666:apis/vhcniphckraatpz7ywxjbmp6ta/types/Subscription/resolvers/onCreateParentsStudents (AWS::AppSync::Resolver) Event Type: update Reason: Resource update cancelled URL: https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-2#/stacks/arn%3Aaws%3Acloudformation%3Aap-southeast-2%3A198433734666%3Astack%2Famplify-teacherapp-dev-134619-apiteacherapp-1DASTOQLVMX22-ParentsStudents-1597ZLBS0Z09S%2Fb5581010-1ed8-11ed-92c3-066fa68469a2/events Resource Name: ParentsStudents-vhcniphckraatpz7ywxjbmp6ta (AWS::DynamoDB::Table) Event Type: update Reason: Resource handler returned message: "Cannot update GSI's properties other than Provisioned Throughput and Contributor Insights Specification. You can create a new GSI with a different name." (RequestToken: e68fba5c-87e9-4245-9271-97e71983c3f1, HandlerErrorCode: InvalidRequest) URL: https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-2#/stacks/arn%3Aaws%3Acloudformation%3Aap-southeast-2%3A198433734666%3Astack%2Famplify-teacherapp-dev-134619-apiteacherapp-1DASTOQLVMX22-ParentsStudents-1597ZLBS0Z09S%2Fb5581010-1ed8-11ed-92c3-066fa68469a2/events 2022-11-14T03:33:56.480Z [INFO]: 2022-11-14T03:33:56.480Z [WARNING]: ✖ An error occurred when pushing the resources to the cloud 2022-11-14T03:33:56.484Z [WARNING]: ✖ There was an error initializing your environment. 2022-11-14T03:33:56.489Z [INFO]: [0mResourceNotReady: Resource is not in the state stackUpdateComplete[0m [0m at constructor.setError (/snapshot/repo/build/node_modules/aws-sdk/lib/resource_waiter.js:182:47)[0m [0m at Request.CHECK_ACCEPTORS (/snapshot/repo/build/node_modules/aws-sdk/lib/resource_waiter.js:44:12)[0m [0m at Request.callListeners (/snapshot/repo/build/node_modules/aws-sdk/lib/sequential_executor.js:106:20)[0m [0m at Request.emit (/snapshot/repo/build/node_modules/aws-sdk/lib/sequential_executor.js:78:10)[0m [0m at Request.emit (/snapshot/repo/build/node_modules/aws-sdk/lib/request.js:686:14)[0m [0m at Request.transition (/snapshot/repo/build/node_modules/aws-sdk/lib/request.js:22:10)[0m [0m at AcceptorStateMachine.runTo (/snapshot/repo/build/node_modules/aws-sdk/lib/state_machine.js:14:12)[0m [0m at /snapshot/repo/build/node_modules/aws-sdk/lib/state_machine.js:26:10[0m [0m at Request.<anonymous> (/snapshot/repo/build/node_modules/aws-sdk/lib/request.js:38:9)[0m [0m at Request.<anonymous> (/snapshot/repo/build/node_modules/aws-sdk/lib/request.js:688:12)[0m [0m at Request.callListeners (/snapshot/repo/build/node_modules/aws-sdk/lib/sequential_executor.js:116:18)[0m [0m at Request.emit (/snapshot/repo/build/node_modules/aws-sdk/lib/sequential_executor.js:78:10)[0m [0m at Request.emit (/snapshot/repo/build/node_modules/aws-sdk/lib/request.js:686:14)[0m [0m at Request.transition (/snapshot/repo/build/node_modules/aws-sdk/lib/request.js:22:10)[0m [0m at AcceptorStateMachine.runTo (/snapshot/repo/build/node_modules/aws-sdk/lib/state_machine.js:14:12)[0m [0m at /snapshot/repo/build/node_modules/aws-sdk/lib/state_machine.js:26:10[0m [0m at Request.<anonymous> (/snapshot/repo/build/node_modules/aws-sdk/lib/request.js:38:9)[0m [0m at Request.<anonymous> (/snapshot/repo/build/node_modules/aws-sdk/lib/request.js:688:12)[0m [0m at Request.callListeners (/snapshot/repo/build/node_modules/aws-sdk/lib/sequential_executor.js:116:18)[0m [0m at callNextListener (/snapshot/repo/build/node_modules/aws-sdk/lib/sequential_executor.js:96:12)[0m [0m at IncomingMessage.onEnd (/snapshot/repo/build/node_modules/aws-sdk/lib/event_listeners.js:335:13)[0m [0m at IncomingMessage.emit (events.js:412:35)[0m [0m at IncomingMessage.emit (domain.js:475:12)[0m [0m at endReadableNT (internal/streams/readable.js:1334:12)[0m [0m at processTicksAndRejections (internal/process/task_queues.js:82:21) {[0m [0m code: 'ResourceNotReady',[0m [0m retryable: false,[0m [0m time: 2022-11-14T03:33:51.471Z,[0m [0m statusCode: 200,[0m [0m retryDelay: 30000[0m [0m}[0m 2022-11-14T03:33:56.509Z [ERROR]: !!! Build failed 2022-11-14T03:33:56.510Z [ERROR]: !!! Non-Zero Exit Code detected 2022-11-14T03:33:56.510Z [INFO]: # Starting environment caching... 2022-11-14T03:33:56.510Z [INFO]: # Uploading environment cache artifact... 2022-11-14T03:33:56.640Z [INFO]: # Uploaded environment cache artifact 2022-11-14T03:33:56.640Z [INFO]: # Environment caching completed Terminating logging...

Expected behavior

When a previous build fails, I expect following builds to work as expected. I would expect my joining table which contains two GSI's to be able to update correctly.

Reproduction steps

Have a graph schema where Amplify generates a join table within DynamoDb View Schema in GraphQl schemas section.
Ensure that your amplify version is set to 9.2.1 and attempt to trigger a build
Build should fail asking you to update to the latest version, go ahead and do that.
After updating the CLI, update the "Live Package Overrides" in build settings to meet 10.4.1
Trigger a build and you should be met with my error.

GraphQL schema(s)

```graphql # Put schemas below this line type Parent @model { id: ID! email: String! firstName: String! lastName: String! prefix: String avatarId: Int! themeColor: String! students: [Student] @manyToMany(relationName: "ParentsStudents") } type Student @model { id: ID! email: String studentName: String! studentCode: String! @index(name: "studentByCode", sortKeyFields: ["createdAt"], queryField: "studentByCode") avatarId: Int! themeColor: String! teacher: Teacher! @belongsTo parents: [Parent] @manyToMany(relationName: "ParentsStudents") currentGoal: Goal @hasOne createdAt: AWSDateTime! } ```

Project Identifier

Project Identifier: 567aa1a8e857d2b2a182bd41487f625c

Log output

``` # Put your logs below this line ```

Additional information

In the past I have had an issue with this which resulted in me deleting my join table and recreating it. I now have valuable data here which links students to parents and can't afford to delete this table. Deleting and recreating the join table is not an option for me.

I pushed a change approximately 2 weeks ago which had no errors so this appears to be new. The only other thing I did was do an amplify pull into my local before triggering my dev CI/CD process.

josefaidt commented 1 year ago

Hey @Taylor-S :wave: thanks for raising this and for including the detailed context! That is certainly helpful. Reading the issue I'm curious if there are any differences in the cli.json file in your project locally vs the copy stored in your deployment bucket?

Taylor-S commented 1 year ago

Thanks for getting back to me @josefaidt , 👋 Just checked the cli.json file on my local machine against the cli.json file (studio-backend_cli.json) within the S3 bucket. They are identical.

{
  "features": {
    "graphqltransformer": {
      "addmissingownerfields": true,
      "improvepluralization": false,
      "validatetypenamereservedwords": true,
      "useexperimentalpipelinedtransformer": true,
      "enableiterativegsiupdates": true,
      "secondarykeyasgsi": true,
      "skipoverridemutationinputtypes": true,
      "transformerversion": 2,
      "suppressschemamigrationprompt": true,
      "showfieldauthnotification": false,
      "securityEnhancementNotification": false
    },
    "frontend-ios": {
      "enablexcodeintegration": true
    },
    "auth": {
      "enablecaseinsensitivity": true,
      "useinclusiveterminology": true,
      "breakcirculardependency": true,
      "forcealiasattributes": false,
      "useenabledmfas": true
    },
    "codegen": {
      "useappsyncmodelgenplugin": true,
      "usedocsgeneratorplugin": true,
      "usetypesgeneratorplugin": true,
      "cleangeneratedmodelsdirectory": true,
      "retaincasestyle": true,
      "addtimestampfields": true,
      "handlelistnullabilitytransparently": true,
      "emitauthprovider": true,
      "generateindexrules": true,
      "enabledartnullsafety": true
    },
    "appsync": {
      "generategraphqlpermissions": true
    },
    "latestregionsupport": {
      "pinpoint": 1,
      "translate": 1,
      "transcribe": 1,
      "rekognition": 1,
      "textract": 1,
      "comprehend": 1
    },
    "project": {
      "overrides": true
    }
  },
  "debug": {
    "shareProjectConfig": false
  }
}

josefaidt commented 1 year ago

Hey @Taylor-S thanks for confirming! Can you describe the changes that were made prior to this failed push attempt? Did it fail to build on 9.2.1 and then fail on the latest CLI version (10.4.1) with a different error?

Build should fail asking you to update to the latest version, go ahead and do that.

And as a side note, what was the messaging from the CLI recommending you to upgrade?

Taylor-S commented 1 year ago

Hey @josefaidt , please see my answers to your questions.

Changes made before the failed push attempt

I made some frontend html changes to my codebase. The deployment process worked as expected. It wasn't until to weeks later where I again made some frontend changes but also did an amplify pull (nothing changed) that the build step in the deployment process failed.

Did it fail to build on 9.2.1 and then fail on the latest CLI version (10.4.1) with a different error?

Yes this is correct. Here is a screenshot of the error from the first attempt (9.2.1) Here's a screenshot of the second error (10.4.1)

And as a side note, what was the messaging from the CLI recommending you to upgrade?

You can see in the first screenshot above. It asked that I update my CLI version as well as ensure the pinned down version was correct after update.

I went back and researched the last time I had this error. That time it was due to a build timeout. To me it looks to be an issue when the build process itself fails. It's like a link or relationship to the GSI's are broken between builds somehow. Not sure if it's a cache or something...honestly I have no clue. Re-running the deployment just fails each time with the same GSI's error.

josefaidt commented 1 year ago

Hey @Taylor-S :wave: thanks for the clarification!

You can see in the first screenshot above. It asked that I update my CLI version as well as ensure the pinned down version was correct after update.

It appears at some point a more recent version of the CLI was used prior to pushing to the environment where Hosting has an older Amplify CLI version set, and is therefore unaware of the newer feature flag showfieldauthnotification. Do you also experience this issue if you remove the feature flag and downgrade back down to 9.x?

Taylor-S commented 1 year ago

Thanks for getting back to me @josefaidt , For more clarification. I am the sole developer working on this project and I haven't pushed with an updated version of the cli until I got the error stating that I try updating to a newer version.

Anyways, here are my results from your reply:

I downgraded the amplify cli locally back to @9.2.1
It appears as though that field had been in my code repository for 3 months and working with 9.2.1 thus far. It was all re-generated upon the amplify push command.
The pushing process continued and again got stuck at my join table having an issue with the GSI's. Just to be clear, I haven't touched these resources outside of Amplify. The only time I have had this issue occur in the past was when the build/push phase of my app's deployment failed due to some reason like a timeout. Could this by an out of sync issue? I believe the screenshot below indicates this may be a possibility? (Sorry if it doesn't, still learning AWS)

I'm pretty keen to learn how to avoid this. Obviously you guys need to investigate and diagnose the issue but is this something I'm doing wrong or is this something on Amplify's end. I'm too afraid to push app updates to my production environment until I feel everything is stable.

Thanks for your help so far :)

Taylor-S commented 1 year ago

HI @josefaidt, Just checking in to see if there's been any findings around this issue?

Taylor-S commented 1 year ago

Hi @josefaidt, Does it look like there will be a fix where I can easily go ahead and successfully build or will I have to remove and re-create my table?

If the latter, should I be following these articles in order to prevent data loss?

Taylor-S commented 1 year ago

Hi @josefaidt Any movement here? Were you able to reproduce the issue in an amplify project?

Mickeel commented 1 year ago

Hi @Taylor-S , thank you for detailing what you faced. We also faced the same problem. Do you happen to have any update of this issue yet?

Taylor-S commented 1 year ago

Hi @Mickeel , Unfortunately, I haven't seen any movement on this issue.

@josefaidt , Have you managed to reproduce this issue in an amplify project and made any progress in remedying this issue?

bergmorten commented 9 months ago

I've the same issue. A build in the console did timeout, which stopped further deployments since amplify thinks another deployment are running. Then I deleted the deployment-state.json in the bucket and try to redeploy. I have also tried amplify pull and then amplify push --force.

Only to get Cannot update GSI's properties other than Provisioned Throughput and Contributor Insights Specification. You can create a new GSI with a different name.

Taylor-S commented 9 months ago

Hey @bergmorten,

I've disbanded my startup project since I couldn't overcome this issue and other challenges, making it difficult to continue. I unfortunately don't have a solution for you. I hope you can find one.

@josefaidt , any progress on replicating and solving this issue on your end?

aws-amplify / amplify-category-api