Open tuliomourazup opened 4 years ago
I had this error when a resource has the same name/prefix as a function. I fixed this by changing the name of the resource
We had the same problem. We added a new lambda function and it caused the circular dependency issue. Renaming the new lambda function fixed the problem.
I am getting a similar issue. The plugin worked fine on the first try. but after getting the following warning -
Serverless: Recoverable error occurred (TooManyRequestsException: Rate exceeded
I added resourceConcurrency(set to 10) to my splitStacks configuration to avoid the above warning. but upon redeploy, I am getting the following error
ServerlessError: The CloudFormation template is invalid: ValidationError: Circular dependency between resources:[GetAllUsersLambdaFunction,....
Below is the configuration I am using with this plugin. is there any workaround available to this?
custom:
splitStacks:
nestedStackCount: 2
perFunction: false
perType: false
perGroupFunction: true
resourceConcurrency: 10
@gautampambhar - I just wasted a day on this. For some reason the resourceConcurrency features adds references to dependsOn
field as soon as you move resources to a nested stack, which is what perGroupFunction does. This is a clear bug to me, about to raise an issue for it.
We have the same issue as described here and https://github.com/dougmoscrop/serverless-plugin-split-stacks/issues/188.
If we have resourceConcurrency
in our config, it does not allow us to deploy the stack for the first time (completely new deployment from scratch).
On the other hand, stackConcurrency
works for the first deployment but does not work for subsequent ones.
To better show these issues, please take a look at the cases below.
The config below is not working if we want to deploy the stack for the first time:
splitStacks:
perFunction: true
perType: false
perGroupFunction: false
stackConcurrency: 5
resourceConcurrency: 10
We have this error:
Circular dependency between resources: [DisconnectIdpIamRoleLambdaExecution, DisconnectIdpLambdaFunction, ApiGatewayMethodUsersDisconnectDashidpOptions, ApiGatewayResourceUsersDisconnectDashidp, ApiGatewayMethodUsersDisconnectDashidpPost, DisconnectIdpLambdaPermissionApiGateway..., ].
Removing resourceConcurrency
from the config above makes deploying the stack for the first time possible.
splitStacks:
perFunction: true
perType: false
perGroupFunction: false
stackConcurrency: 5
However, the above config is not working for subsequent deployments. We have the same error as we have when trying to deploy the stack for the first time with resourceConcurrency
set.
The CloudFormation template is invalid: Circular dependency between resources: [UpdateEmailTemplateNestedStack, CustomMessageNestedStack, SetDefaultEmailTemplateNestedStack, UpdateGroupNestedStack, ...].
It looks like resourceConcurrency
creates some circular dependencies between resources and stackConcurrency
does the same but between nested stacks (if that makes sense?).
Is there any way to handle this issue? Currently, we removed both options and left only:
splitStacks:
perFunction: true
perType: false
perGroupFunction: false
However, I am afraid that we will hit API rate limit errors in the nearest feature and the problem will arise again.
@gautampambhar, @sweepy84, @vicary, maybe, you found some solution to this problem? Thank you for any tips in advance! 🙏
@turakvlad tl;dr craft your own stack-mapping.js in logical layers to prevent them from stacking on each others.
There are limitations of a fully managed SAM template composition. There is no easy fix and you are required to be concise about all the resources being created, and how are they allocated into sub-stacks.
The most probable cause from our experience is when the following resources for the same lambda being sliced into different nested stacks:
It is compounded for the 2nd deployment because existing resources are not allowed to move between nested stacks by default, although you may force it to move it's still not scalable.
Try to allocate your resources by type and dependency into it's own "stack chain", for example we have 4 layers of stack chains:
The ideas is that resources in a layer will never try to depends on it's own layer or lower layers.
Bare with the abstract descriptions because there is really no way to exhaustively explain CloudFormation, it's a pain and you kinda have to dig into those generated SAM templates in JSON in the directory .serverless/
and really understand your own stack.
EDIT
We never use resourceConcurrency
because CloudFormation tries to create resources as a top-down dependency graph, resources will be created once it's dependency is ready. It is already the quickest way possible.
CloudFormation has built-in rate limit backoff and retry mechanism. If you encounter rate limits, it's already a result after multiple retries. If you see them, you should contact your account manager or create a support ticket.
Hint: It's not the first time we see an internal bug where CloudFormation is out of sync with the underlying resources' creation status. So don't be afraid to ask them for help.
@vicary, I have no idea how I missed your answer!
Thank you very much for such a detailed response. We will definitely try this approach and craft our own stack-mapping.js
.
Error: The CloudFormation template is invalid: Circular dependency between resources: [ApiGatewayMethodCategoriesCategoryidVarOptions, ApiGatewayMethodHospitalsHospitalidVarMedicinesGet, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarEquipmentsEquipmentidVarOptions, ApiGatewayMethodHospitalsHospitalidVarCleanOptions, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarStaffsGet, ApiGatewayMethodUsersListallGet, ApiGatewayResourceHospitalsHospitalidVarEpi, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarEquipmentsOptions, ApiGatewayMethodHospitalsPost, ApiGatewayResourceMedicinesMedicineidVarVolumesMedicinevolumeidVar, ApiGatewayMethodHospitalsHospitalidVarMedicinesMedicineidVarPost, ApiGatewayMethodUsersHospitalidVarPut, ApiGatewayMethodMedicinesMedicineidVarVolumesMedicinevolumeidVarOptions, ApiGatewayMethodFederationDashstatesFederationstateidVarCitiesGet, ApiGatewayMethodTermStatusGet, ApiGatewayMethodHospitalsHospitalidVarPendenciesAreasMovementsOptions, ApiGatewayMethodMedicinesMedicineidVarVolumesGet, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarPut, ApiGatewayResourceHospitalsHospitalidVar, ApiGatewayMethodUsersHospitalidVarUserGet, ApiGatewayResourceHospitalsHospitalidVarMedicinesMedicineidVarVolumes, ApiGatewayMethodUsersByemailOptions, ApiGatewayMethodHospitalsHospitalidVarEpiEpiidVarPost, ApiGatewayAuthorizer, ApiGatewayMethodCategoriesCategoryidVarGet, ApiGatewayMethodApidocsGet, ApiGatewayResourceHospitalsHospitalidVarAreasAreaidVarMovements, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarMovementsDelete, ApiGatewayMethodHospitalsHospitalidVarOptions, ApiGatewayResourceHospitalsHospitalidVarEquipments, ApiGatewayMethodTableauOptions, ApiGatewayResourceHospitalsHospitalidVarMedicinesMedicineidVarInactivate, ApiGatewayMethodHospitalsProfileOptions, ApiGatewayResourceMedicinesMedicineidVarVolumes, ApiGatewayMethodHospitalsHospitalidVarEquipmentsMovementsGet, ApiGatewayMethodUsersHospitalidVarGet, ApiGatewayMethodUserDashmanagementDisassociateDashuserHospitalidVarPut, ApiGatewayResourceHospitalsHospitalidVarAreasAreaidVar, ApiGatewayMethodHospitalsHospitalidVarInactivatePut, ApiGatewayMethodUsersGet, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarGet, ApiGatewayMethodMedicinesHospitalsGet, ApiGatewayResourceHospitalsHospitalidVarAreasAreaidVarStaffs, ApiGatewayMethodHospitalsHospitalidVarInactivateDashuserOptions, ApiGatewayMethodHospitalsHospitalidVarEquipmentsEquipmentidVarMovementsPost, ApiGatewayResourceHospitalsHospitalidVarAreasAreaidVarEquipmentsEquipmentidVar, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarEquipmentsGet, ApiGatewayMethodProfilesOptions, ApiGatewayResourceFederationDashstatesFederationstateidVarCities, ApiGatewayResourceHospitalsProfile, ApiGatewayResourceUserDashmanagementResetpass, ApiGatewayMethodFederationDashstatesOptions, ApiGatewayMethodHospitalsHospitalidVarMedicinesMedicineidVarVolumesOptions, ApiGatewayMethodUsersLoggeduserinfoOptions, ApiGatewayResourceHospitals, ApiGatewayMethodHospitalsHospitalidVarMedicinesMedicineidVarOptions, ApiGatewayResourceHospitalsHospitalidVarAreasAreaidVarShiftsShiftidVar, ApiGatewayResourceCategories, ApiGatewayResourceMedicinesHospitals, ApiGatewayResourceProfiles, ApiGatewayMethodHospitalsHospitalidVarPendenciesMedicinesMovementsGet, ApiGatewayMethodUsersPut, ApiGatewayMethodHospitalsHospitalidVarPendenciesEquipmentsMovementsOptions, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarEquipmentsEquipmentidVarGet, ApiGatewayResourceHospitalsHospitalidVarEpiEpiidVar, ApiGatewayResourceHospitalsHospitalidVarMedicinesMedicineidVar, ApiGatewayResourceHospitalsListall, ApiGatewayMethodMedicinesHospitalsOptions, ApiGatewayMethodHospitalsHospitalidVarEpiGet, ApiGatewayResourceHospitalsHospitalidVarClean, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarMovementsOptions, ApiGatewayMethodHospitalsProfileGet, ApiGatewayResourceUsersHospitalidVarUser, ApiGatewayResourceHospitalsHospitalidVarMedicines, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarDelete, ApiGatewayMethodHospitalsHospitalidVarPendenciesEpiMovementsOptions, APINestedStack, ApiGatewayMethodUsersLoggeduserinfoGet, ApiGatewayMethodHospitalsHospitalidVarEquipmentsGet, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarEquipmentsEquipmentidVarPost, ApiGatewayMethodCategoriesGet, ApiGatewayMethodUserDashmanagementInactiveOptions, ApiGatewayMethodProfilesTypesGet, ApiGatewayMethodHospitalsListallGet, ApiGatewayResourceFederationDashstates, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarOptions, ApiGatewayMethodUsersPost, ApiGatewayResourceHospitalsHospitalidVarAreasAreaidVarShifts, ApiGatewayResourceUsersHospitalidVar, ApiGatewayResourceUsersListall, ApiGatewayMethodHospitalsHospitalidVarPendenciesStaffsMovementsGet, ApiGatewayResourceProfilesTypes, ApiGatewayMethodUserDashmanagementAssociateDashuserHospitalidVarOptions, ApiGatewayResourceUserDashmanagement, ApiGatewayResourceMedicinesMedicineidVar, ApiGatewayMethodHospitalsHospitalidVarPendenciesMedicinesMovementsOptions, ApiGatewayResourceUserDashmanagementAssociateDashuserHospitalidVar, ApiGatewayResourceHospitalsHospitalidVarInactivateDashuser, ApiGatewayMethodFederationDashstatesFederationstateidVarCitiesOptions, ApiGatewayMethodUserDashmanagementOptions, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarStaffsOptions, ApiGatewayMethodProfilesTypesOptions, ApiGatewayMethodHospitalsHospitalidVarEquipmentsEquipmentidVarOptions, ApiGatewayResourceHospitalsHospitalidVarAreas, ApiGatewayMethodMedicinesMedicineidVarVolumesOptions, ApiGatewayMethodTableauGet, ApiGatewayMethodHospitalsHospitalidVarEquipmentsEquipmentidVarMovementsOptions, ApiGatewayMethodApidocsOptions, ApiGatewayMethodHospitalsHospitalidVarEquipmentsMovementsOptions, ApiGatewayMethodHospitalsHospitalidVarAreasOptions, ApiGatewayMethodHospitalsHospitalidVarMedicinesMedicineidVarInactivatePut, ApiGatewayResourceUserDashmanagementInactive, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarShiftsShiftidVarOptions, ApiLambdaPermissionApiGateway, ApiGatewayMethodSystemStatusTpsOptions, ApiGatewayMethodHospitalsHospitalidVarAreasPost, ApiGatewayMethodUsersByemailGet, ApiGatewayResourceUsersByemail, ApiGatewayMethodHospitalsHospitalidVarCleanDelete, ApiGatewayResourceHospitalsHospitalidVarAreasAreaidVarEquipments, ApiGatewayMethodFederationDashstatesGet, ApiNestedStack, ApiGatewayMethodHospitalsOptions, ApiGatewayMethodHospitalsHospitalidVarMedicinesOptions, ApiGatewayMethodUsersHospitalidVarUserOptions, ApiGatewayMethodHospitalsHospitalidVarPendenciesEpiMovementsGet, ApiGatewayMethodUsersListallOptions, ApiGatewayMethodHospitalsHospitalidVarMedicinesMedicineidVarVolumesPost, ApiGatewayMethodHospitalsHospitalidVarAreasGet, ApiGatewayMethodProfilesGet, ApiGatewayResourceUsers, ApiGatewayMethodTermStatusOptions, ApiGatewayMethodHospitalsHospitalidVarPendenciesEquipmentsMovementsGet, ApiGatewayResourceFederationDashstatesFederationstateidVar, ApiGatewayMethodHospitalsHospitalidVarPendenciesStaffsMovementsOptions, ApiGatewayResourceUserDashmanagementAssociateDashuser, ApiGatewayMethodTermGet, ApiGatewayResourceSystemStatus, ApiGatewayResourceTerm, ApiGatewayMethodUsersHospitalidVarOptions, ApiGatewayMethodHospitalsHospitalidVarGet, ApiGatewayResourceTableau, ApiGatewayMethodCategoriesOptions, ApiGatewayMethodHospitalsGet, ApiGatewayMethodUserDashmanagementDisassociateDashuserHospitalidVarOptions, ApiGatewayMethodTermPost, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarShiftsShiftidVarPost, ApiGatewayMethodMedicinesMedicineidVarVolumesMedicinevolumeidVarGet, ApiGatewayResourceApidocs, ApiGatewayMethodUserDashmanagementInactiveDelete, ApiGatewayMethodHospitalsHospitalidVarInactivateOptions, ApiGatewayRestApi, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarMovementsPut, ApiGatewayMethodUserDashmanagementResetpassOptions, ApiGatewayMethodMedicinesMedicineidVarGet, ApiGatewayMethodUserDashmanagementAssociateDashuserHospitalidVarPost, ApiGatewayMethodHospitalsHospitalidVarEpiOptions, ApiGatewayMethodHospitalsHospitalidVarEpisCleanOptions, ApiGatewayMethodSystemStatusTpsGet, ApiGatewayResourceSystemStatusTps, ApiGatewayMethodHospitalsHospitalidVarEquipmentsEquipmentidVarPut, ApiGatewayResourceCategoriesCategoryidVar, ApiGatewayMethodHospitalsHospitalidVarEpisCleanDelete, ApiGatewayResourceUserDashmanagementDisassociateDashuserHospitalidVar, ApiGatewayResourceHospitalsHospitalidVarInactivate, ApiGatewayResourceTermStatus, ApiGatewayMethodUserDashmanagementResetpassPut, ApiGatewayMethodUserDashmanagementPost, ApiGatewayResourceHospitalsHospitalidVarEquipmentsEquipmentidVar, ApiGatewayMethodMedicinesMedicineidVarOptions, ApiGatewayMethodHospitalsHospitalidVarEpiEpiidVarOptions, ApiGatewayMethodHospitalsHospitalidVarPut, ApiGatewayMethodHospitalsListallOptions, ApiGatewayMethodHospitalsHospitalidVarMedicinesMedicineidVarInactivateOptions, ApiGatewayResourceUsersLoggeduserinfo, ApiGatewayResourceMedicines, ApiGatewayMethodUsersOptions, ApiGatewayMethodHospitalsHospitalidVarPendenciesAreasMovementsGet, ApiGatewayMethodHospitalsHospitalidVarAreasAreaidVarMovementsGet, ApiGatewayResourceSystem, ApiGatewayResourceUserDashmanagementDisassociateDashuser, ApiGatewayMethodHospitalsHospitalidVarEquipmentsOptions, ApiGatewayMethodTermOptions, ApiGatewayMethodHospitalsHospitalidVarEpiEpiidVarGet, ApiGatewayMethodHospitalsHospitalidVarInactivateDashuserPut]