investigate running more than 200 tasks on one scheduler

[belforte]

• it is an old limit, current hardware may support more
• as far as Stefano knows, it is controlled by this in /etc/condor/config.d/82_cms_schedd_crabschedd_generic.config

  START_SCHEDULER_UNIVERSE = (TotalSchedulerJobsRunning < 200) && ((ShadowsRunning is undefined) || (ShadowsRunning < .95*$(MAX_JOBS_RUNNING)))

• Stefano tried, and failed, to increase by adding a modified START_SCHEDULER_UNIVERSE in /etc/condor/config.d/99_local_tweaks.config with

  START_SCHEDULER_UNIVERSE = (TotalSchedulerJobsRunning < 250) && ((ShadowsRunning is undefined) || (ShadowsRunning < .95*$(MAX_JOBS_RUNNING)))

TO DO

• [ ] find why adding to 99_local_tweaks failed even after schedd stop/start given that

  belforte@vocms0106/config.d> condor_config_val START_SCHEDULER_UNIVERSE
  (TotalSchedulerJobsRunning < 250) && ((ShadowsRunning is undefined) || (ShadowsRunning < .95*15000))

• [ ] missing that. Try by editing directly 82_cms_schedd_crabschedd_generic

• [ ] overload one production scheduler with real work and see if we can push further

• [ ] decide on new policy if any, knowing that

  • a larger limit may give more flexibility to deal with short-lived crisis *having too many tasks active at same time will make them run slower as they compete for CPU