Open tartachyov opened 1 year ago
tartachyov
commented
1 year ago Tried to downgrage to Amplify CLI 10.5.2 and run amplify push --allow-desctrictive-changes --force but after successfull roll-back having the same result during deployment of changes with error "table name should be passed"
AnilMaktala
commented
1 year ago Hey @tartachyov, can you try this command "amplify push --allow-desctrictive-changes --force --debug" and share us the console logs.
tartachyov
commented
1 year ago Unfortunatelly not anymore, because I was able to deploy my changes and my amplify project is not being in this error state anymore. What I can clearly say, that the origin of that issue was related to one particular change in my schema by adding new @index to existing object.
I added following to my schema, which looks pretty standard and I did this many times before.
type ObjA @model {
id: ID!
allBsOfA: [ObjB] @hasMany(indexName: "objBByAId", fields: ["id"])
}
type ObjB @model {
id: ID!
objAId: ID @index(name: "objBByAId")
objA: ObjA @belongsTo(fields: ["objAId"])
}Also when I downgraded version of amplify-cli to 9.x.x (one of the latest from 9.x.x – 9.2.1 or so) there was a different output message comparing to 10.x.x saying something like "Waiting for DynamoDB inidices..." which helped me to identify the problem. That message was hanging for infinite amount of time and I had to abort amplify push command.
chrisbonifacio
commented
1 year ago @tartachyov thank you for the update and happy to hear you were able to get unblocked!
tartachyov
commented
1 year ago Hi there! This issue is still relevant, because I cannot add @index directive to my existing schema. "table name should be passed" erroe message always appears when I try to add @index. Please reopen this ticket. And this issue is urgent to our project. Are you able to resolve it?
tartachyov
commented
1 year ago @chrisbonifacio please keep me posted!
tartachyov
commented
1 year ago Is it related to backfilling of DynamoDB indexes? https://docs.amplify.aws/cli/graphql/troubleshooting/#deploying-multiple-index-changes-at-once
tartachyov
commented
1 year ago Is it related to backfilling of DynamoDB indexes? https://docs.amplify.aws/cli/graphql/troubleshooting/#deploying-multiple-index-changes-at-once
No, it does not look to be aplicable and required here at all.
Also not clear what is the content of ddb_to_es.py in that example.
tartachyov
commented
1 year ago Just FYI!
I can clearly see that amplify push --allow-destructive-graphql-schema-updates --force deployment failed with error "table name should be passed",
I can also se that in CloudFormation dashboard the amplify env stack's status is UPDATE_COMPLETE and last deployment time is a while ago in the past (not latest attempt),
but !!! in DynamoDB dashboard inside table "Indexes" tab I can see that my new index was created successfully and its size 0 with count 0 objects.
How could this be? is it related to partially deployed env changes? Probably this is the only explanation, but anyway whenever I run amplify push --allow-destructive-graphql-schema-updates --force it's going to roll-back previous partially deployed changeset, so it should not generate any conflicts.
tartachyov
commented
1 year ago So, then what is the problem?! Looking forward for any relevant solution proposal...
tartachyov
commented
1 year ago Guys! Do something! ASAP!
Because after trying to add @index and seeing error, now I cannot do any change to the schema at all even without adding @index! I cannot even redeploy previous version with no changes, because of seeing the same error again "table name should be passed". My amplify env seems to be completely dead, I cannot do anything with it at all.
Ok, I tried to do the same thing in a separate env and here is what I found. I was able to add @index with no issues exactly the same way as I tried to do in a primary env before, great!
But then I decided to remove it and everything was also fine until the point when amplify push unexpectedly exited without finishing deployment with no error or anything just in the middle (see the screenshot 1). And now no matter what I'm trying to do: adding index, removing index, redeploying previous successfully deployed version, everything ends with error "table name should be passed" (see screenshot 2).
So, seems that after crashing amplify push deployemnt in the middle env is stuck in weird state.
--force flag does not help as well as amplify push --iterative-rollback. The same error again and again with no additional info.
Result: Both envs are completely dead and do not respond. In a meantime in CloudFormation status "UPDATE_COMPLETE" is with timestamp from previous successful deployment.
WTF!?
P.S. I was using latest Amplify CLI version (10.8.0 and 10.8.1 yesterday and today, maybe also one version 10.x.x before) all this time since reopening this ticket. Maybe I'm going to downgrade at least to get more meaningful output error message...
tartachyov
commented
1 year ago I tried once again with another env!
But this time I decided to downgrade to Amplify CLI 9.2.1. Additionally this time instead of doing several changes to schema I decided to do one little change.
Previously it was removing one index, modifying another and adding couple more which makes deployment to have multiple sequential parts ("Deploying 1 of 3...").
But this time only one little change which is "remove one @index" that's it, very simple!
And here is what I've got during deployment (screenshot 1). And now again I cannot do anything with this env. Both commands amplify push --iterative-rollback --allow-destructive-graphql-schema-updates and amplify push --allow-destructive-graphql-schema-updates --force result the same error (screenshot 2)
And again env status is UPDATE_COMPLETE!
So, I have 3 corrupted envs so far...
tartachyov
commented
1 year ago @AnilMaktala @chrisbonifacio any updates about it? Any solution proposals? Looking forward, thanks!
AnilMaktala
commented
1 year ago Hey @tartachyov, We apologize for the inconvenience. Please try the command "amplify push --allow-destructive-graphql-schema-updates --force --debug" and share the console logs with us.
chrisbonifacio
commented
1 year ago @tartachyov oh wow, I completely misunderstood your comment and closed this out prematurely. My apologies.
Reading through the info you provided, I'm curious what steps you took to reproduce this issue in a new environment. I deployed the schema you shared with ObjA & ObjB models, first push succeeded. I then removed the @index directive and simplified the relationship to:
type ObjA @model {
id: ID!
allBsOfA: [ObjB] @hasMany
}
type ObjB @model {
id: ID!
objA: ObjA @belongsTo
}However, the second push succeeded so I don't think I reproduced this accurately.
Can you list the states of your schema as you were deploying until you came across this issue? I also have a feeling that the functions in your amplify app might have something to do with the error.
alharris-at
commented
1 year ago It appears this error message is thrown from the iterative deployment engine, this means that for some reason, one invocation of this method is accepting a list of table names with a null or undefined value being passed in the array somewhere https://github.com/aws-amplify/amplify-cli/blob/b01aa4baf90d6426bbcd4ef9c5603a3ea697c09c/packages/amplify-provider-awscloudformation/src/iterative-deployment/deployment-manager.ts#L412.
Marking this as a bug, and we'll need to get a reliable reproduction of the issue to mitigate and test fix.
tartachyov
commented
1 year ago Yes, probably this is an origin of the issue! And btw this is still important blocker for our project, looking forward for solution! I would say it's P1
parvusville
commented
1 year ago Also facing this after adding a single new @key. Before this I pushed other changes, including new fields (including an @connection field to an existing index) and auth rules.
First time pushing locally, it failed with no error message. I did amplify push --iterative-rollback, then tried pushing again and now I got the table name should be passed error. I also see that this new index has been created to my model in DDB.
Doing amplify push --iterative-rollback seems to work for me, but then if I try to add any @key at all I get the table name should be passed error. This is severely blocking the development.
parvusville
commented
1 year ago Hello, just checking in if there's anything we could do to help you investigate this issue. I'm happy to jump onto a call, or provide my full schema if it helps.
tartachyov
commented
1 year ago @parvusville I'm just creting new amplify environment from scratch and then migrate all the data and users into it every time I need to modify indexes.
parvusville
commented
1 year ago @tartachyov That is not feasible solution for us, as it introduced way too much hustle in a production environment. However I am interested in hearing what is your process for migrating the data and users?
parvusville
commented
1 year ago It seems I have found a workaround and a possible cause to this issue. After removing plenty of non-necessary field level @auth rules (#1703 ), I am now able to push indexes again. Every field level @auth rule do show up on the API build process on amplify push, maybe this has something to do with having too many?
ndaba1
commented
3 months ago I was able to fix this by deleting the deployment-state.json file in the deployment bucket on s3
meteordefect
commented
3 months ago We are now experiencing this error multiple times. AWS support are unable to replicate it as they cannot create a schema as big as ours. What work is being done to rectify this issue?
meteordefect
commented
3 months ago It seems like AWS support are not able to replicate many issues that users face when the environment has a very large/complex schema. Can you AWS support please build a large schema (approaching the cfn limit) so that they can begin testing issues like this?
ikzhr
commented
2 months ago @dpilch
I had the same problem, this error broke my amplify env (can not amplify push anymore)
It may take time to resolve the root cause, so I want to at least make sure that this error does not break the amplify env.
The direct cause of this error is that falsy values can be added to tableNames at the following code.
https://github.com/aws-amplify/amplify-cli/blob/main/packages/amplify-provider-awscloudformation/src/graphql-resource-manager/amplify-graphql-resource-manager.ts#L192
I think if an exception is thrown at this point, the env is not expected to break because the process will stop before the deployment process begins. Is this idea reasonable?
How did you install the Amplify CLI?
No response
If applicable, what version of Node.js are you using?
No response
Amplify CLI Version
10.6.2
What operating system are you using?
macOS
Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.
No, I did no changes directly to the underlying resources managed by Amplify.
Describe the bug
I did some changes in schema.graphql and in functions and then running
amplify pushas usual faced error "table name should be passed".Project Identifier: 66937b402f25cb91155bb86fb029afed
P.S. for some reason amplify CLI detected changes across all resources, while in reality few little changes were applied. Might be because CLI version was recently updated.
Expected behavior
Expected behavior that updated schema successfully deployed.
Reproduction steps
amplify pushProject Identifier
66937b402f25cb91155bb86fb029afed
Log output
Additional information
No response
Before submitting, please confirm: