sopel
commented
11 years ago This is the list as of my notes, I'll check them once solved or extracted to a dedicated issue:
- [x] 1. It turns out that Route 53 doesn't replace existing records, so the first stack creation failed due to the target
Arecordlogsearch.cityindextest5.co.ukbeing in use already- we've solved this by deploying a record with the stack name instead and converting
logsearch.cityindextest5.co.ukinto aCNAME, which is actually more flexible anyway (see below); requires documentation of this manual step and/or a mini CFN stack for this still (I think I prefer that approach due to the implied notifications and audit trail) - see #154
- we've solved this by deploying a record with the stack name instead and converting
- [x] 2. The former would allow a new stack to be deployed in parallel and switched to via DNS only, which is a common way to allow rollback (and also a/b testing in advanced scenarios, which R53 actually supports as well, see Creating Weighted Resource Record Sets)
- this is currently blocked due to the EBS data volume handling, so we might look into a workaround to shorten the update time at least (low priority)
- see #155
- [x] 3. The initial spot instance creation failed two times due to the odd way Auto Scaling seems to handle this (complex, will elaborate elsewhere), it's no issue once the stack is up though.
- I initially thought we need to increase the
WaitConditiontimeout to provide some wiggle room, but it turns out I simply miscalculated the related event durations and the existing 900 seconds are sufficient in fact - regardless, this should be documented and/or accounted for in a more formal way to avoid future surprises
- see #156
- I initially thought we need to increase the
- [x] 4. Another DNS issue has been on file already, but not filed on an issue yet, insofar the tier refactoring yields
redisas the new DNS name for the shipping target, the formerbrokeris decommissioned.- see #157
- [x] 5. A second shipper issue has been the outdated SSH key, so the shipper couldn't connect
- @mrdavidlaing has deployed the current one has been deployed during cluster redeployment already, see https://github.com/cityindex/logsearch-development-flow/issues/130#issuecomment-23396391
- [x] 6. The CloudWatch instrumentation choked due a case change in embedded properties like
name=>Name- this is strange, insofar the former stack worked flawless with the lower case variation, but probably not worth investigating outside of a quick peek.- see #158
- [x] 7. The two Elasticseach nodes came up in an unexpected order, yielding the ephemeral one becoming the master rather than the EBS one.
- see #159
- [x] 8.The Loggly log configuration required manual addition of the new nodes, which defeats auto scaling at least.
- see #160
- [x] 9. The Librato dashboard had been mostly configured against dedicated instances too.
- it is now using
allsources, which currently works, but would fail if deploying a second cluster for example - a better approach would be to use a Dynamic Source for this
- see #161
- it is now using
- [x] 10. The
aws s3command failed to execute with weird errors, which resulted in the post install script not running in turn.- it has been manually executed after cluster redeployment, so all dedicated nodes are reporting correctly right now, but the Auto Scaling group is not anymore after scaling activities (likewise the dedicated nodes would fail to report after a CloudFormation update of the instance type for example)
- see #162
A recent complete cluster redeployment uncovered a few minor but partially odd regressions regarding the envisioned self-contained stack usage via a single CloudFormation template call, which are going to be shortly documented and assessed here in subsequent comments and handled via dedicated issues in case.