Open mook-as opened 5 years ago
We have created an issue in Pivotal Tracker to manage this:
https://www.pivotaltracker.com/story/show/168061254
The labels on this github issue will be updated when the story is started.
@mook-as ,thanks!yes,we noticed this problem and tracked through https://github.com/cloudfoundry/app-autoscaler-release/issues/207. A PR https://github.com/cloudfoundry/app-autoscaler-release/pull/209 is raised to resolve this issue, but still under review.
Furthermore, @mook-as , we did the force termination as well to reproduce the issue, but do you happen to notice any other approach to trigger the failure?
The liquibase related pre-start job is there for a quite long time, but it never failed in bosh or scf deployment previously...
If "force termination" is the only trigger, I am quite curious which change in scf will kill on pre-start job ... Any cue from your side?
Yeah, I'm not sure why we're seeing forced termination (actually, I think it's just that the DB connection died); I suspect it's just us bumping cf-deployment leading to a larger footprint in the rest of the deployment, overloading the system slightly and making the autoscaler bits slower?
Hi there!
We're getting an error on startup, where autoscaler-metrics can't start because liquibase attempts to acquire a lock, but its locking implementation means there's a chance of it dangling with no owner:
Steps to reproduce:
(This was on SCF, but as far as I can tell this is an issue with liquibase, not how you run it.)
autoscaler-metrics
pod.)Starting Liquibase
, forcibly terminate the process.Details
It appears that liquibase implements locking by inserting a row in a database, instead of taking a database lock (row-level or table-level). See this stackoverflow question for a similar situation, and recovery where manual intervention in the DB is required. I think it might actually be trying to use transactions, though; just removing the "lock" got it to recover.
I'm not sure what you could do to fix this (other than trying to fix it upstream in liquibase — unfortunately my Java is terrible). But I figured I should file it at the minimum.
Thanks!