Open namrathaurs opened 7 months ago
Discussed my findings/observations with Marco and following are his inputs:
reserve
that overcommits the number of glideins [link here]. The tag named idle_glideins_per_entry
has documentation for max
but not reserve
.Initially, we thought this might be something on the GlideinWMS factory and this could be happening due to a bug. After the investigation, it seems more likely that DE could be requesting for more glideins. A suggestion provided was to thoroughly review DE code to avoid this scenario.
The very first observation of this was during some activity on a Decision Engine (DE) instance that talks to an ITB factory (798, running GlideinWMS 3.10.5-1). The glideins were being requested even though the job that was submitted by the DE had completed. Verified to ensure that the requests were not coming from either the DE client in question or some other clients because of jobs being in the respective job queues. A request coming from the client includes two numbers:
ReqMaxGlideins
andReqIdleGlideins
which are of interest to understand the underlying behavior. Upon further investigation of theglideclient
andglidefactoryclient
classads, it was found that:condor_q
shows jobs in running and idle state (5 processes submitted and each one has a sleep for 10 minutes):condor_q
:After excessively requesting glideins, at some point, the
glideclient
classad vanishes from the factory after which no more glideins are requested in the factory. Since this classad vanishes after its expiration, glideins not being requested makes sense since the classad is no longer present.This very same behavior of glideins being requested even though the DE job queue is empty was also observed in a couple of instances:
source1
went offline. Onlytest_channel
was steady butresource_request
went offline. This further confirmed that it is something on the DE side.