Circular Dependency Issue in our Amplify CI/CD pipeline

[dhruvbansal2]

Amplify CLI Version

10.5.0

Question

Hello!

I am running into a circular dependency issue in our Amplify CI/CD pipeline when attempting to push the following schema changes

type Service @model {
   id: ID!
   ownerID: ID
   owner: Owner @hasOne(fields: ["personID"])
}

type Car @model {
   owners: [Owner!] @hasMany(indexName: "byCar", fields: ["id])
}

type Owner @model {
   id: ID!
   carID: ID! @index(name: "byCar")
   car: Car! @belongsTo(fields: ["carID"])
}

Error:

UPDATE_FAILED  apiwww  AWS::CloudFormation::Stack Tue Dec 13 2022 02:22:36 GMT+0000 
(Coordinated Universal Time) Circular dependency between resources: [Service, Car, SearchableStack, Owner, ConnectionStack, CustomResourcesjson ...]

This doesn't happen in my developer environment when I do amplify push -y, however, it keeps failing in our CI/CD pipeline. I noticed that the amplifyPush --simple script has been deprecated - could this issue be caused by that change?

We do have other models in the schema with similar dependencies that were introduced earlier this year and that doesn't seem to have any problems building out in the pipeline.

[dhruvbansal2]

Hello! Any ideas of what could be causing this?

This is preventing us from introducing new schema into our production system and blocking the release of some critical features.

[SilverLinings89]

It seems like we have a similar problem. We are currently migrating from 10.3.1 (Amplify CLI) to 10.5.1 and have problems deploying because of circular dependencies, which also list a SearchableStack. For context: We have introduced @searchable very recently and the circular dependency seems to be closely related with @searchable / SearchableStack. If we remove @ searchable the error disappears and from what it looks like, the circular dependency is simply listing all databases/models we have.

[hanna-becker]

I'm on the same project as @SilverLinings89. We were able to work around the issue for now by downgrading to amplify-cli version 10.4.1. As he said, we were only able to get rid of the circular dependency error by removing @searchable and amplify force-pushing. Then, when trying to re-add the annotation the issue would come back. Downgrading and force pushing fixed things for us. Looking at the release log, my guess would be this regression may have been introduced by the searchable mock work released in 10.5.0.

[hanna-becker]

@dhruvbansal2 Is it possible your CI build settings are set to use the latest amplify-cli version?

[dhruvbansal2]

@hanna-becker thanks for the info!

We manually upgraded recently to 10.5.0 (and set it specifically to Amplify CLI - 10.5.0 in the AWS Amplify Console in build settings).

We saw this issue after adding new models (without the @searchable directive). Even before the upgrade we had models that had @searchable directive and upgrading our cli version to 10.5.0 didn't seem to cause them to break. It only breaks (in the Amplify pipeline) when I add a new model (& their associated connections)

I'm testing a few different approaches, including going down to 10.4.1.

[hanna-becker]

Well, it was actually similar for us. We added our @searchable on the cli version 10.3.1. After upgrading to 10.5.1 (which deployed the already existing state just fine), trying to add anything to the backend resulted in the cyclic dependency error. Even just adding a Hello World lambda function.

[dhruvbansal2]

Yea we are noticing if we just change an auth rule (for certain models), our pipeline fails with the same circular dependency.

Downgrading to 10.4.1 isn't even an option for us since, for whatever reason, our pre-push hooks don't run on that version (I think there was a bug related to that as well). We would need to go all the way down to cli version 9 to get this to work.

@josefaidt any recommendations on how else we can workaround this issue?

[hanna-becker]

@josefaidt Is there a fix planned for this? We'd hate being stuck on amplify cli 10.4.1 for a long time.

[hamaf]

I also tested Amplify CLI 10.6.1 and the Circular dependency between resources persists.

[josefaidt]

Hey @dhruvbansal2 and @hanna-becker apologies for the delay here. The searchable issue was resolved in Amplify CLI 10.6.2, and I am unable to reproduce this issue with Amplify CLI 10.7.0. Do you still experience this issue after upgrading to 10.6.2 or 10.7.0?

[isaackehle]

Hey @josefaidt , we reverted back to v9.2.1 when this was an active blocking issue for us. Soon after, 10.6.2 was released. We have not yet put the time back in to try to upgrade further.

[hanna-becker]

Hi @josefaidt, thanks for getting back. We upgraded to Amplify 10.7.1 today and had no issues with circular dependencies when adding a new lambda function afterwards. However, we also removed the @searchable annotation for other reasons 2 weeks ago. So I cannot tell if the issue is resolved, in case it had a correlation with @searchable, as we suspected.

[PeteDuncanson]

I've been getting this on and off for the last 2 days.

I've updated to v10.71.

Seems if I do one schema change and push that up it might be happy with it but then follow up pushes to CI fail with the Circular Reference error and a listing of all my table names.

Releasing at the minute is a odd game of chance as a result.

I've done a amplify pull to ensure my environment is up to date but still it comes and goes. Naturally making development tricky and confidence rocked.

[PeteDuncanson]

Ok so I've changed my build settings to use 10.4.1 for now and I've had a few deployments go through with no issues. For now I'll stick to that version until I hear otherwise in here that its been fixed.

Screenshot of where to set it for those that follow:

[sundersc]

@dhruvbansal @hanna-becker - Could you try this tagged version and let me know if this solves your problem?

npm i -g @aws-amplify/cli@10.5.3-circular-dep-fix.0

[PeteDuncanson]

@sundersc I've been running that version on my CI for the last week and no issues at all with it :)

So whatever you did seems it might have worked.

Any idea when I can upgrade yet?

[sundersc]

@PeteDuncanson Thanks for confirming. PR# 1305 includes this fix.

[BmjEarhart]

Hi there, our team has been following this issue as we've been seeing the very same issue, a circular dependency error listing all of our resources starting from when we upgraded to 10.6.x.

We tracked the issue back to this ticket and found that the 10.5.3-circular-dep-fix.0 tagged version solved our issues perfectly for many weeks, until today. As of this morning we haven't been able to run a deployment without getting the circular dependency error, with the only changes being a fairly benign front-end change in our code.

We've tried

1) deploying with 10.5.3-circular-dep-fix.0, 2) deploying our most recent code with the latest amplify CLI version 3) re-deploying old code versions that have previously succeeded with 10.5.3-circular-dep-fix.0 again,

And all seem failing with the same error: Circular dependency between resources: [ *every resource type, as well as SearchableStack*]. And again, we're only seeing this via CI - "amplify push" from desktop seems to work as expected

Is there anything that could have changed recently to explain this? And is anyone else experiencing this issue returning recently?

[djorgji]

We have the same issue right now as @BmjEarhart, the whole team is dead in the water unable to continue working. It seems like no matter which CLI version we configure in the CICD it wont work. (AWS::CloudFormation::Stack), Event Type: create, Reason: Circular dependency between resources: [ *every resource type, as well as, ConnectionStack, CustomResourcesjson ]

We had been pinned to 10.2.3 for months now, all of a sudden this morning our PR-Previews started failing, we thought maybe upgrading CLI would help, so we upgraded to 11.0.2 still the same error, reverted back to 10.2.3 still the same issue, tried 10.5.3-circular-dep-fix.0 no difference. The whole day has been spent trying to figure out this issue...

Because we have connected branches we can't even publish via CLI from our local computers. Deploying with the CLI (amplify push --yes) from out computers works.

Please advise.

[isaackehle]

Our team has been stuck using 9.2.1 since we first noticed this issue. I'm curious if anyone else has gone the headless script route, of if you are still using amplifyPush --simple

[sundersc]

Could you try with 11.0.3? This includes the fix https://github.com/aws-amplify/amplify-category-api/pull/1305 released with the 10.5.3-circular-dep-fix.0 tagged version.

[BmjEarhart]

So for today we've seen these outcomes:

• One CI deployment with CLI overridden to 10.5.3-circular-dep-fix.0 succeeded with code it previously failed on
• One CI deployment without any CLI override failed with the circular dependency error
• One CI deploy with CLI overridden explicitly to 11.0.3 succeeded.

I'm not sure how to check what version gets used when no override is set, but I suppose it must have been an older version than 11.0.3. We're running some more test deployments to see whether it consistently succeeds with 11.0.3.

Still unclear why the circular-dep-fix stopped working then recovered, but I'll report here if we see any more issues with 11.0.3

[djorgji]

Somehow miraculously 11.0.3 worked (still unsettling) for us the day after my post. Haven't tried any further experiments as I do not feel comfortable rocking the boat. We will probably be pinning to this version for a while.

[zirkelc]

Same issue here event with version 11.0.3.

[naedx]

I upgraded to 11.0.3. I now have this issue.

[naedx]

I was able to use the stack resources to track down the conflicts. I found that there were resolvers that were referenced in the pipeline of sibling stacks (eg. createdBy, createdAt), which I think is due to Amplify applying deduplication to the resolvers.

I modified those fields' authorization configurations and that was enough to separate them and I was able to push after that.

[sundersc]

@zirkelc @naedx - I can see that resolver deduping could be the cause of the problem but I'm unable to reproduce a case where it produces circular dependency between child stacks. If you have your project in an error state, could you please send your project's amplify folder to amplify-cli@amazon.com?

[zirkelc]

@sundersc I have downgraded to v10.x and was able to push again. Then I upgraded again to v11.0.3 and don't have an issue at the moment. I'll let you know when it happens again.

[KidEma]

Has this fix worked for everyone? We are still having this issue on v11.0.4 with "DisableResolverDeduping": true. Deployments work when making any random changes in the schema (even adding comments works), but they fail with this same error otherwise.

[SaurabBajgai]

Could you try with 11.0.3? This includes the fix #1305 released with the 10.5.3-circular-dep-fix.0 tagged version.

Same issue with v11.1.1 even with version 11.0.3 issue persists

[SaurabBajgai]

Has this fix worked for everyone? We are still having this issue on v11.0.4 with "DisableResolverDeduping": true. Deployments work when making any random changes in the schema (even adding comments works), but they fail with this same error otherwise.

We have the same issue right now as @KidEma ,on latest version i.e v11.1.1 Deployment works on any random changes in the schema file (even adding comments works), but they fail with this same error otherwise. @sundersc

[josefaidt]

Hey folks, re-opening this per the last few comments. There was a fix that was identified and released with @aws-amplify/amplify-category-api@5.2.1 and will appear in an Amplify CLI release in the near future

[phani-srikar]

The latest fix that @josefaidt pointed out related to resolver de-duplication should be available in CLI v12.0.0 or later.

[ashrielbrian]

@josefaidt @phani-srikar i'm encountering the same issues as those above. Using 12.0.3 in our amplify console's CI pipeline - is the fix you mentioned already pushed in 12.0.3? I can't find any references to amplify-category-api@5.2.1 in the release notes.

Also, is "DisableResolverDeduping": true still necessary? If so, do we manually add that in transform.conf.json?

[ObinnaAka]

This is the strangest Amplify bug I have ever found. I have spent 3 days trying to fix it, trying everything from amplify push --force , to turning of deduplication, deleting deployment-state.json, and finally destroying my env, then spinning up an entirely new one. Nothing worked

For some context, this is what my schema looked like

type PageComment
    @model
    @auth(
        rules: [
            { allow: public, operations: [read] }
            { allow: private, operations: [read, create] }
            {
                allow: owner
                ownerField: "owner"
                operations: [read, create, update, delete]
            }
            { allow: groups, groups: ["Admin"], operations: [read, update, delete] }
        ]
    ) {
    id: ID!
    title: String
    content: String
    creator: User @belongsTo

    owner: String
        @auth(
            rules: [
                { allow: private, operations: [create] }
                { allow: owner, operations: [read, delete] }
                { allow: groups, groups: ["Admin"], operations: [read, update, delete] }
            ]
        )

    type: PageCommentType
    visibility: Visibility
    position: PagePostion

    starterPacResource: StarterPacResource @belongsTo

    createdAt: AWSDateTime!
    updatedAt: AWSDateTime!

    data: [String]
    aiData: [String]
    tags: [Tag] @hasMany
}

And the error I got when trying to do a deployment was

circular dependency between resources: [ConnectionCtack, CustomResourcesJson, PageComment]

I debugged by looking through my cloudformation stack and all templates to see if there was a circular dependency. There was none, to the best of my knowledge.

But I got it fixed. I removed the PageComment model from the schema; then I made a small change by adding this.

type Test {
    id: ID!
    name: String
}

I deployed that, and it went through successfully. Then I tried deploying again, and it finally went through.

However, even though it is working, this is EXTREMELY unsettling. The only way I could fix this was by deleting a database and then adding it back in. That is extremely concerning for a production environment. I hope this gets addressed and fixed soon.

[mangBert]

I have the same issue - did all the suggested solution but nothing works. Does it needs to delete the table? This is concerning as we have prod env as well.

[ndaba1]

I was experiencing the same issue. For me it appeared after adding a field level auth rule, specifically on the owner field. Once I removed it, it stopped failing

[ObinnaAka]

New thing that solved it for me, removed these fields from the offending object.

createdAt: AWSDateTime! updatedAt: AWSDateTime!

[matt-at-allera]

I'm running into this issue on Amplify cli version 12.11.1. Disabling deduplication is solving the issue for me. However, we pretty consistently are hit by AppSync throttling during CloudFormation updates, since the deduplication turned off bloats the number of resolvers in our API. Has anyone found a consistent workaround with deduplication enabled?

[acrolink]

AWS with its huge bills AND stupid issues like this !! Wasted 2 hours on THIS !! Solution was to downgrade to v12.11.1. I will never use AMPLIFY on any future project. Not first issue with Amplify.