aws-khargita
commented
6 months ago Thank you for opening this bug report.
We have identified the issue to be the parallel creation of multiple custom resources each attempting to create the same log stream at once. This results in a ResourceAlreadyExistsException.
Once the log stream exists, further invocations of the custom resources lambda handler will not try to recreate it, which would explain the success on retry. Since a new log stream is created daily, this problem will resurface on a daily basis.
A fix for this bug will be included in the upcoming release. For now, when using cloudformation templates to manage schedules, you can use the DependsOn attribute to allow one custom resource to run synchronously prior to the others. This should prevent the race condition.
Example of what that would look like:
Resources:
SampleSchedule1:
Type: 'Custom::ServiceInstanceSchedule'
Properties:
ServiceToken: !Ref ServiceInstanceScheduleServiceTokenARN #do not edit this line
NoStackPrefix: 'False'
Name: my-renamed-sample-schedule
Description: a full sample template for creating cfn schedules showing all possible values
Timezone: America/New_York
Enforced: 'True'
Hibernate: 'True'
RetainRunning: 'True'
StopNewInstances: 'True'
UseMaintenanceWindow: 'True'
SsmMaintenanceWindow: 'my_window_name'
Periods:
- Description: run from 9-5 on the first 3 days of March
BeginTime: '9:00'
EndTime: '17:00'
InstanceType: 't2.micro'
MonthDays: '1-3'
Months: '3'
- Description: run from 2pm-5pm on the weekends
BeginTime: '14:00'
EndTime: '17:00'
InstanceType: 't2.micro'
WeekDays: 'Sat-Sun'
SampleSchedule2:
Type: 'Custom::ServiceInstanceSchedule'
Properties:
ServiceToken: !Ref ServiceInstanceScheduleServiceTokenARN #do not edit this line
NoStackPrefix: 'True'
Description: a sample template for creating simple cfn schedules
Timezone: Europe/Amsterdam
Periods:
- Description: stop at 5pm every day
EndTime: '17:00'
DependsOn: SampleSchedule1
FugroEgger
CrypticCabub
Describe the bug
Very similar to this bug that was preciously closed: https://github.com/aws-solutions/instance-scheduler-on-aws/issues/339
When I use 'Custom::ServiceInstanceSchedule' to create schedules I intermittently get the CloudFormation error 'Invalid PhysicalResourceId', and a retry of the deployment works the second time.
I also occasionally get this cloud formation error 'CREATE_FAILED | Received response status [FAILED] from custom resource.'
To Reproduce
To reproduce simply deploy, update, and delete custom schedules via Cloudformation and eventually you will get this error out of the blue. Most of the time it works and sometimes it doesn't.
I am currently working with a customer that deploys the Scheduler Hub along side the IAC schedules in one template via a pipeline. In this case it will fail 75% of the time and work only after retrying.
Expected behavior
My customer would expect the Custom Resource creation to work every time consistently
Please complete the following information about the solution:
[v1.5.3 ] Version: [e.g. v1.0.0]
[ us-east-1] Region: [e.g. us-east-1]
[ Not in any meaningful way, I get this error even deploying the exact template] Was the solution modified from the version published on this repository?
[ ] If the answer to the previous question was yes, are the changes available on GitHub?
[Yes ] Have you checked your service quotas for the sevices this solution uses?
[ Will add if available] Were there any errors in the CloudWatch Logs? Troubleshooting
Screenshots attached
