Open bparees opened 6 years ago
@mfojtik we're still seeing this a lot in our extended test runs. Any suggestions?
/cc @derekwaynecarr
/cc @liggitt @deads2k
@mfojtik we're still seeing this a lot in our extended test runs. Any suggestions?
"Still" or "just started again". What percentage of jobs are failing on it?
@stevekuznetsov I've got master and node metrics, but not controller metrics. Where is the script that describes what to gather?
@deads2k i've only started looking into this again but the last 3 times i ran our extended tests, I saw this in several test failures for each run. so, 100% over the last few days, that i've looked at.
As to whether there was ever a period in recent history where it wasn't happening, i'm not sure.
(i'm also not sure why our extended jobs would be particularly vulnerable to it, vs conformance jobs)
@deads2k i've only started looking into this again but the last 3 times i ran our extended tests, I saw this in several test failures for each run. so, 100% over the last few days, that i've looked at.
Is there any sane way for you to see if it spiked about two weeks ago? I re-sliced some startup code that seemed to significantly improve our normal CI, but if it suddenly started spiking, that's where I'd be starting my search.
What do you mean by "controller metrics"? We dump pprof
output but I'm not sure we do anything from Prometheus?
Is there any sane way for you to see if it spiked about two weeks ago?
not really. our extended test jobs have been a mess for a month and a half due to storage issues and devmapper issues and I don't really want to try to weed through that.
Issues go stale after 90d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle stale
.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen
.
If this issue is safe to close now please do so with /close
.
/lifecycle stale
Stale issues rot after 30d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle rotten
.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen
.
If this issue is safe to close now please do so with /close
.
/lifecycle rotten /remove-lifecycle stale
@smarterclayton heh......
/remove-lifecycle rotten /lifecycle frozen
@deads2k @mfojtik @smarterclayton indicated he has a bug open for this (issues w/ the service account controller getting bogged down)
https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17092/test_pull_request_origin_extended_conformance_install/2451/
The default timeout for this is 2 minutes.