Proxy bugs idler a lot!

[concaf]

Every time someone calls jenkins.openshift.io, proxy jumps into action and tried to un-idle the jenkins pod. Here is how it does it -

1. Once token_json is set by auth, it un-idles, sets the cookies, and redirects to self to remove token_json from the URL - https://github.com/fabric8-services/fabric8-jenkins-proxy/blob/0c4f75bfc810aaf18291645da3aad8d946fc9c8f/internal/proxy/ui_requests.go#L64-L76

2. After being redirected, instantaneously, proxy finds the cookies and un-idles again and calls processTemplate() - https://github.com/fabric8-services/fabric8-jenkins-proxy/blob/0c4f75bfc810aaf18291645da3aad8d946fc9c8f/internal/proxy/ui_requests.go#L129-L160

3. The template sends a request to self yet again, which means calling proxy again and calling idler again - https://github.com/fabric8-services/fabric8-jenkins-proxy/blob/0c4f75bfc810aaf18291645da3aad8d946fc9c8f/static/html/index.html#L29-L32

This means that's 3 (or maybe more undiscovered) un-idle requests from proxy. This sort of messes up the logs, and probably causes max retries to be reached in idler making it unreliable for un-idling.

[concaf]

This coincides with the idler logs -

{"component":"openshift-client","level":"info","msg":"Un-idling jenkins in sminglan-jenkins","time":"2018-05-24T12:06:14Z"}
{"component":"openshift-client","level":"info","msg":"Un-idling content-repository in sminglan-jenkins","time":"2018-05-24T12:06:14Z"}
{"attempt":"(3/3)","component":"user-idler","id":"8800be34-a6df-405c-84f5-6b214050b13a","level":"info","msg":"About to un-idle jenkins, Reason: DoneBuild BuildName:unidletest-1 Last:2018-05-08 10:01:34 +0000 UTC","time":"2018-05-24T12:06:14Z","username":"sminglan"}
{"component":"openshift-client","level":"info","msg":"Un-idling jenkins in sminglan-jenkins","time":"2018-05-24T12:06:14Z"}
{"attempt":"(3/3)","component":"user-idler","id":"8800be34-a6df-405c-84f5-6b214050b13a","level":"info","msg":"About to un-idle content-repository, Reason: DoneBuild BuildName:unidletest-1 Last:2018-05-08 10:01:34 +0000 UTC","time":"2018-05-24T12:06:14Z","username":"sminglan"}
{"component":"openshift-client","level":"info","msg":"Un-idling content-repository in sminglan-jenkins","time":"2018-05-24T12:06:14Z"}
{"component":"user-idler","id":"8800be34-a6df-405c-84f5-6b214050b13a","level":"warning","msg":"Skipping un-idle request since max retry count has been reached.","time":"2018-05-24T12:06:14Z","username":"sminglan"}
{"component":"user-idler","id":"8800be34-a6df-405c-84f5-6b214050b13a","level":"warning","msg":"Skipping un-idle request since max retry count has been reached.","time":"2018-05-24T12:06:14Z","username":"sminglan"}
{"level":"info","msg":"Jenkins is already starting/running on sminglan-jenkins","time":"2018-05-24T12:06:15Z"}
{"level":"info","msg":"Jenkins is already starting/running on sminglan-jenkins","time":"2018-05-24T12:06:16Z"}

[concaf]

ping @kishansagathiya @sthaha @chmouel

[sthaha]

@kishansagathiya @containscafeine I have a solution for this already in progress as a part of solving the 504 issue. Will submit a patch that addresses just this part soon.

[sthaha]

This sort of messes up the logs, and probably causes max retries to be reached in idler making it unreliable for un-idling.

Could you expand on that? How does this make un-idling - unreliable? Shouldn't idler be able to process mutiple unilde request for the same namespace and only process the first request and ignore the rest of them?

[concaf]

How does this make un-idling - unreliable?

You're correct because DoUnidle() is not being called whenever the unidle API is being hit, so it's fine; I speculated this while debugging https://github.com/fabric8-services/fabric8-jenkins-idler/issues/233, but this is probably not the root cause