search

dmwm / CRABServer

16 stars 38 forks source link

Overflow via JobRouter edits too many jobs #8802

Open belforte opened 15 hours ago

belforte commented 15 hours ago

There is ample evidence that jobs which should not be routed, are instead routed. Also there are jobs routed from European T1's which according to our monitoring run in the US. We have disabled JobRouter daemon, but need to fix.

As a start will put JobRouter in "tagging mode": add a new classAd with the indication of where job would be routed, but do not edit DESIRED_Sites.

belforte commented 15 hours ago

tag mode enabled in puppet qa via https://gitlab.cern.ch/ai/it-puppet-hostgroup-vocmsglidein/-/merge_requests/288 now deployed on vocms059 note: puppet agent will not star Job_Router daemon. Needs and explicit sudo condor_on --daemon Job_Router

belforte commented 15 hours ago

and the problem is immediately very clear, the table for the first job (KIT) has been applied to all of the following jobs, which were in Idle since some time, but targeting very different sites !!

[root@vocms059 config.d]# condor_q -con HasBeenOverflowRouted -af:h JobUniverse jobstatus desired_sites desired_sites_orig new_sites
JobUniverse jobstatus desired_sites             desired_sites_orig        new_sites                      
5           1         T1_DE_KIT                 T1_DE_KIT                 T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T1_ES_PIC                 T1_ES_PIC                 T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T1_FR_CCIN2P3             T1_FR_CCIN2P3             T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T1_IT_CNAF                T1_IT_CNAF                T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T1_RU_JINR                T1_RU_JINR                T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T1_UK_RAL                 T1_UK_RAL                 T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T1_US_FNAL                T1_US_FNAL                T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T2_CH_CERN_HLT            T2_CH_CERN_HLT            T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T2_CH_CERN_P5             T2_CH_CERN_P5             T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T2_IN_TIFR                T2_IN_TIFR                T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T2_IT_Rome                T2_IT_Rome                T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T2_LB_HPC4L               T2_LB_HPC4L               T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T2_PK_NCP                 T2_PK_NCP                 T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T2_TR_METU                T2_TR_METU                T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T2_UK_SGrid_Bristol       T2_UK_SGrid_Bristol       T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T3_BG_UNI_SOFIA           T3_BG_UNI_SOFIA           T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T3_IN_TIFRCloud           T3_IN_TIFRCloud           T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T3_IT_Opportunistic_dodas T3_IT_Opportunistic_dodas T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T3_MX_Cinvestav           T3_MX_Cinvestav           T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T3_TW_TIDC                T3_TW_TIDC                T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T3_US_FNALLPC             T3_US_FNALLPC             T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T3_US_Ookami              T3_US_Ookami              T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T3_US_Test                T3_US_Test                T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
5           1         T3_US_UMD                 T3_US_UMD                 T1_DE_KIT,T2_DE_DESY,T2_DE_RWTH
[root@vocms059 config.d]#
belforte commented 13 hours ago

side notes, as I read with more care the current JR config I realize that

belforte commented 10 hours ago

I tried various simplifications and changes to () in an attempt to "fix the requirements", but no matter what I tried, all idle jobs are routed via the same route !

I wrote to HTC support list

Dear experts,

we (CMS-CRAB) have used JobRouter since years to edit queued jobs in CMS global pool to redirect then from busy sites to sites with possibly available slots, something that we call "overflow". Details of why and what happens to those jobs when they run do not matter, simply to introduce the name.

Alas we have been using the deprecated JOB_ROUTER_ENTRIES_CMD macro where custom script of ours was called to create routing on the fly So last month we rewrote our stuff using static routing table in the configuration file reproducing the same desiderata. But once put in production things started to go nuts and we had to disable.

Unfortunately we have no experience with the new configuration, and nobody else in CMS or glideinWms is using it.

We are running $CondorVersion: 23.9.6 2024-08-08 BuildID: 748275 PackageID: 23.9.6-1 GitSHA: dfdd9eaa $
and followed the example in https://htcondor.readthedocs.io/en/latest/grid-computing/job-router.html defining a list of mutually exclusive routes as in [1]

But once the daemon starts the first route is matched to all possible jobs, even if the Requirements = (DESIRED_SITES=="sitename") is not satisfied.

Some things work as expected i.e. the edit in place, attribute setting, and the route listed first in JOB_ROUTER_ROUTE_NAMES is clearly the one used first. But that first rout is also applied to jobs where DESIRED_SITES is a different string.

Basically all my idle jobs get the same value of NEW_SITES and I can change that by changing the order or routes in JOB_ROUTER_ROUTE_NAMES [2].

Can you spot something which we do wrong here ?

I also noted that while jobs are routed, the RoutedBy attributes is not set (ref https://htcondor.readthedocs.io/en/latest/admin-manual/configuration-macros.html#JOB_ROUTER_SOURCE_JOB_CONSTRAINT and https://htcondor.readthedocs.io/en/latest/admin-manual/configuration-macros.html#JOB_ROUTER_NAME )

Let us know if there's any more information which I can send.

Thanks!!!

Stefano

[1]

[root@vocms059 config.d]# pwd
/etc/condor/config.d
[root@vocms059 config.d]# cat 90_jobrouter.config
# Configuration file for the JobRouter
#
JOB_ROUTER_NAME = OverflowRouter

JOB_ROUTER_SOURCE_JOB_CONSTRAINT = ((JobUniverse==5) && (jobstatus==1))

# Static route names for each T1
JOB_ROUTER_ROUTE_NAMES =  T1_DE_KIT T1_IT_CNAF T1_UK_RAL T1_ES_PIC T1_FR_CCIN2P3

JOB_ROUTER_ROUTE_T1_DE_KIT @=rtkit
  Name = ["Overflow:T1_DE_KIT"](overflow:T1_DE_KIT)
  EditJobInPlace = True
  Requirements = (DESIRED_SITES=="T1_DE_KIT")
  SET NEW_SITES "T2_DE_DESY"
  SET HasBeenOverflowRouted True
@rtkit

JOB_ROUTER_ROUTE_T1_ES_PIC @=rtpic
  Name = ["Overflow:T1_ES_PIC"](overflow:T1_ES_PIC)
  EditJobInPlace = True
  Requirements = (DESIRED_SITES=="T1_ES_PIC")
  SET NEW_SITES "T2_ES_CIEMAT"
  SET HasBeenOverflowRouted True
@rtpic

JOB_ROUTER_ROUTE_T1_FR_CCIN2P3 @=rtin2p3
  Name = ["Overflow:T1_FR_CCIN2P3"](overflow:T1_FR_CCIN2P3)
  EditJobInPlace = True
  Requirements = (DESIRED_SITES=="T1_FR_CCIN2P3")
  SET NEW_SITES "T2_FR_GRIF,T2_FR_IPHC"
  SET HasBeenOverflowRouted True
@rtin2p3

JOB_ROUTER_ROUTE_T1_IT_CNAF @=rtcnaf
  Name = ["Overflow:T1_IT_CNAF"](overflow:T1_IT_CNAF)
  EditJobInPlace = True
  Requirements = (DESIRED_SITES=="T1_IT_CNAF")
  SET NEW_SITES "T2_IT_Pisa,T2_IT_Rome"
  SET HasBeenOverflowRouted True
@rtcnaf

JOB_ROUTER_ROUTE_T1_UK_RAL @=rtral
  Name = ["Overflow:T1_UK_RAL"](overflow:T1_UK_RAL)
  EditJobInPlace = True
  Requirements = (DESIRED_SITES=="T1_UK_RAL")
  SET NEW_SITES "T2_UK_London_IC,T2_UK_SGrid_RALPP"
  SET HasBeenOverflowRouted True
@rtral

# How often to poll the job queue to route jobs
JOB_ROUTER_POLLING_PERIOD = 5*60

# Start the Job Router
DAEMON_LIST = $(DAEMON_LIST) JOB_ROUTER
[root@vocms059 config.d]#

[2]

belforte@vocms059/HTCondor> condor_q -con HasBeenOverflowRouted -af:h jobstatus desired_sites  new_sites
jobstatus desired_sites             new_sites
1         T1_DE_KIT                 T2_DE_DESY
1         T1_ES_PIC                 T2_DE_DESY
1         T1_FR_CCIN2P3             T2_DE_DESY
1         T1_IT_CNAF                T2_DE_DESY
1         T1_UK_RAL                 T2_DE_DESY
1         T1_US_FNAL                T2_DE_DESY
1         T2_CH_CERN_HLT            T2_DE_DESY
1         T2_CH_CERN_P5             T2_DE_DESY
1         T2_IN_TIFR                T2_DE_DESY
1         T2_IT_Rome                T2_DE_DESY
1         T2_LB_HPC4L               T2_DE_DESY
1         T2_PK_NCP                 T2_DE_DESY
1         T2_TR_METU                T2_DE_DESY
1         T2_UK_SGrid_Bristol       T2_DE_DESY
1         T3_BG_UNI_SOFIA           T2_DE_DESY
1         T3_IN_TIFRCloud           T2_DE_DESY
1         T3_IT_Opportunistic_dodas T2_DE_DESY
1         T3_MX_Cinvestav           T2_DE_DESY
1         T3_TW_TIDC                T2_DE_DESY
1         T3_US_FNALLPC             T2_DE_DESY
1         T3_US_Ookami              T2_DE_DESY
1         T3_US_Test                T2_DE_DESY
1         T3_US_UMD                 T2_DE_DESY
1         T1_IT_CNAF                T2_DE_DESY
1         T1_IT_CNAF                T2_DE_DESY
1         T1_ES_PIC                 T2_DE_DESY
1         T1_FR_CCIN2P3             T2_DE_DESY
1         T1_UK_RAL                 T2_DE_DESY
1         T1_UK_RAL                 T2_DE_DESY
belforte@vocms059/HTCondor>