search

dmwm / WMCore

Core workflow management components for CMS.
Apache License 2.0
45 stars 106 forks source link

WorkQueue fails to pull work on drain sites #4006

Closed dballesteros7 closed 11 years ago

dballesteros7 commented 11 years ago

If there is a site in draining state, the local WorkQueue will query couch with an URI including all the associated workflows to the draining sites which will often generate the following error:

CouchError - reason: Request-URI Too Large

We would need to split those URIs by workflow, or some reasonable chunks.

dballesteros7 commented 11 years ago

Just to keep record of an example when this is a problem:

2012-08-15 18:17:58,056:ERROR:WorkQueueManagerWorkPoller:Error in work pull loop: NEW ERROR STATUS! UPDATE CMSCOUCH.PY!: 414 CouchError - reason: Request-URI Too Large, data: {'wfs': '["spinoso_TOP-Summer12_FS52-00041_R1599_B3_FSIM_23_LHE_120808_165121_3754", "spinoso_TOP-Summer12_FS52-00164_R1599_B3_FSIM_77_LHE_120808_165508_379", "spinoso_EXO-Summer11-01455_R1571_B106_26_LHE_120721_155457_1258", "etorassa_BPH-Summer12-00029_batch137_v1120711_211903_6758", "etorassa_HIG-Summer12-00809_batch135_v1__120720_155430_5689", "etorassa_HIG-Summer11-01102_batch104_v1120706_120146_9623", "spinoso_HIG-Summer12-00870_R1569_B140_01_LHE_120720_153611_6146", "etorassa_SUS-Summer12-00029_batch139_v1120712_123054_8408", "spinoso_TOP-Summer12_FS52-00056_R1599_B3_FSIM_38_LHE_120808_165227_1425", "spinoso_HIG-Summer12-00872_R1569_B140_03_LHE_120720_153615_7564", "spinoso_EXO-Summer11-01435_R1571_B106_06_LHE_120721_155414_7995", "spinoso_TOP-Summer12-00135_R1598_B142_10_LHE_120725_200857_1861", "spinoso_EXO-Summer12-01737_R1608_B145_38_LHE_120725_201449_7193", "spinoso_SUS-Summer12_FS52-00014_R1618_B5_FSIM_05_LHE_120801_190312_9816", "spinoso_HIG-Summer11-01176_R1640_B111_04_LHE_120808_100219_6506", "spinoso_EXO-Summer12-01684_R1567_B138_25_LHE_120720_104128_4374", "etorassa_HIG-Summer12-00804_batch135_v1__120720_155411_1230", "spinoso_HIG-Summer11-01179_R1640_B111_07_LHE_120808_100225_8890", "casarsa_HIG-Summer12-00890_144_v1120723_112517_2643", "spinoso_TOP-Summer12_FS52-00052_R1599_B3_FSIM_34_LHE_120808_165208_7037", "spinoso_TOP-Summer12_FS52-00031_R1599_B3_FSIM_13_LHE_120808_165038_4018", "etorassa_HIG-Summer11-01095_batch103_v1120706_120000_8858", "etorassa_BPH-Summer12-00066_batch135_v1__120720_155356_6917", "etorassa_BPH-Summer12-00036_batch135_v1120720_155202_7107", "spinoso_EXO-Summer12-01708_R1608_B145_09_LHE_120725_201342_2679", "etorassa_HIG-Summer11-01098_batch103_v1120706_120019_3373", "spinoso_EXO-Summer12-01614_R1523_B126_01_LHE_120713_163324_2493", "spinoso_EXO-Summer11-01459_R1571_B106_30_LHE_120721_155504_5532", "spinoso_EXO-Summer12-01621_R1524_B127_03_LHE_120713_163800_5211", "spinoso_SUS-Summer12_FS52-00010_R1618_B5_FSIM_02_LHE_120801_190307_2951", "spinoso_SUS-Summer12_FS52-00007_R1599_B3_FSIM_04_LHE_120801_184526_9859", "casarsa_HIG-Summer12-00877_143_v1__120720_181205_4479", "spinoso_EXO-Summer11-01431_R1571_B106_02_LHE_120721_155405_4079", "spinoso_TOP-Summer12_FS52-00043_R1599_B3_FSIM_25_LHE_120808_165130_7030", "casarsa_HIG-Summer12-00824_143_v1120720_181117_3331", "casarsa_BPH-Summer12-00074_143_v1120720_181012_3581", "spinoso_EXO-Summer12-01716_R1608_B145_17_LHE_120725_201358_8440", "etorassa_BPH-Summer12-00071_batch139_v2__120712_123041_2228", "spinoso_EXO-Summer11-01445_R1571_B106_16_LHE_120721_155435_7265", "spinoso_SUS-Summer11-00090_R1571_B106_32_LHE_120721_155509_5686", "spinoso_EXO-Summer11-01448_R1571_B106_19_LHE_120721_155441_4443", "spinoso_TOP-Summer12_FS52-00061_R1599_B3_FSIM_43_LHE_120808_165248_2866", "spinoso_TOP-Summer12_FS52-00049_R1599_B3_FSIM_31_LHE_120808_165154_5979", "etorassa_BPH-Summer12-00057_batch135_v1120720_155320_6142", "spinoso_SUS-Summer12_FS52-00008_R1599_B3_FSIM_05_LHE_120801_184529_7486", "spinoso_EXO-Summer11-01439_R1571_B106_10_LHE_120721_155423_6720", "spinoso_EXO-Summer12-01727_R1608_B145_28_LHE_120725_201427_8063", "spinoso_EXO-Summer11-01446_R1571_B106_17_LHE_120721_155437_6912", "etorassa_BPH-Summer12-00059_batch135_v1120720_155328_2555", "etorassa_EWK-Summer12-00085_batch137_v2__120711_212052_9285", "spinoso_EXO-Summer11-01453_R1571_B106_24_LHE_120721_155452_3166", "etorassa_HIG-Summer12-00806_batch135_v1120720_155419_1328", "spinoso_EXO-Summer12-01671_R1567_B138_12_LHE_120720_104101_738", "etorassa_BPH-Summer12-00062_batch135_v1120720_155340_2122", "casarsa_HIG-Summer12-00811_143_v1__120720_181020_7260", "spinoso_TOP-Summer12_FS52-00029_R1599_B3_FSIM_11_LHE_120808_165028_3383", "etorassa_HIG-Summer11-01100_batch104_v1120706_120138_6217", "spinoso_EXO-Summer11-01430_R1571_B106_01_LHE_120721_155403_9155", "casarsa_HIG-Summer12-00820_143_v1120720_181058_7721", "casarsa_HIG-Summer12-00879_143_v1__120720_181217_4151", "spinoso_TOP-Summer12_FS52-00154_R1599_B3_FSIM_67_LHE_120808_165429_1856", "spinoso_TOP-Summer12_FS52-00139_R1599_B3_FSIM_53_LHE_120808_165329_9992", "spinoso_SUS-Summer12_FS52-00005_R1599_B3_FSIM_02_LHE_120801_184521_5078", "casarsa_HIG-Summer12-00888_144_v1120723_112508_5595", "spinoso_TOP-Summer12_FS52-00067_R1599_B3_FSIM_49_LHE_120808_165313_2991", "spinoso_TOP-Summer12-00132_R1598_B142_07_LHE_120725_200851_2744", "spinoso_TOP-Summer12-00134_R1598_B142_09_LHE_120725_200855_1356", "spinoso_EXO-Summer12-01617_R1523_B126_04_LHE_120713_163331_2661", "spinoso_TOP-Summer12_FS52-00145_R1599_B3_FSIM_58_LHE_120808_165352_3812", "etorassa_HIG-Summer12-00807_batch135_v1120720_155423_4302", "spinoso_TOP-Summer12_FS52-00047_R1599_B3_FSIM_29_LHE_120808_165146_7068", "spinoso_EXO-Summer11-01437_R1571_B106_08_LHE_120721_155419_9160", "spinoso_SUS-Summer12_FS52-00004_R1599_B3_FSIM_01_LHE_120801_184519_2096", "spinoso_TOP-Summer12_FS52-00035_R1599_B3_FSIM_17_LHE_120808_165055_7301", "etorassa_BPH-Summer12-00047_batch135_v1__120720_155244_948", "spinoso_EXO-Summer12-01713_R1608_B145_14_LHE_120725_201352_9142", "spinoso_EXO-Summer12-01615_R1523_B126_02_LHE_120713_163325_9696", "spinoso_EXO-Summer12-01674_R1567_B138_15_LHE_120720_104107_5462", "etorassa_BPH-Summer12-00065_batch135_v1120720_155353_1799", "spinoso_EXO-Summer11-01451_R1571_B106_22_LHE_120721_155447_9053", "etorassa_BPH-Summer12-00024_batch147_v4120725_102720_3906", "spinoso_SUS-Summer12_FS52-00011_R1618_B5_FSIM_03_LHE_120801_190309_3660", "spinoso_EXO-Summer11-01452_R1571_B106_23_LHE_120721_155449_8328", "etorassa_BPH-Summer12-00044_batch135_v1__120720_155233_3308", "spinoso_EXO-Summer12-01711_R1608_B145_12_LHE_120725_201348_2314", "etorassa_BPH-Summer12-00054_batch135_v1120720_155309_5510", "spinoso_EXO-Summer12-01616_R1523_B126_03_LHE_120713_163329_4020", "spinoso_TOP-Summer12_FS52-00065_R1599_B3_FSIM_47_LHE_120808_165304_5950", "spinoso_TOP-Summer12_FS52-00044_R1599_B3_FSIM_26_LHE_120808_165133_6356", "etorassa_BPH-Summer12-00039_batch135_v1120720_155214_9073", "etorassa_BPH-Summer12-00035_batch135_v1__120720_155159_5575", "spinoso_SUS-Summer11-00089_R1571_B106_31_LHE_120721_155507_8370", "spinoso_EXO-Summer11-01434_R1571_B106_05_LHE_120721_155413_5560", "spinoso_TOP-Summer12_FS52-00146_R1599_B3_FSIM_59_LHE_120808_165355_7084", "spinoso_EXO-Summer11-01458_R1571_B106_29_LHE_120721_155502_2114", "spinoso_TOP-Summer12_FS52-00169_R1599_B3_FSIM_82_LHE_120808_165527_7041", "spinoso_TOP-Summer12-00127_R1598_B142_04_LHE_120725_200843_9809", "spinoso_EXO-Summer12-01672_R1567_B138_13_LHE_120720_104102_9014", "spinoso_EXO-Summer11-01438_R1571_B106_09_LHE_120721_155421_7066", "spinoso_EXO-Summer12-01622_R1524_B127_04_LHE_120713_163802_2678", "spinoso_EXO-Summer12-01663_R1567_B138_04_LHE_120720_104044_45", "spinoso_EXO-Summer11-01447_R1571_B106_18_LHE_120721_155439_6277", "spinoso_EXO-Summer11-01436_R1571_B106_07_LHE_120721_155416_9718", "spinoso_TOP-Summer12_FS52-00157_R1599_B3_FSIM_70_LHE_120808_165442_552", "spinoso_TOP-Summer12_FS52-00163_R1599_B3_FSIM_76_LHE_120808_165504_3033", "etorassa_BPH-Summer12-00050_batch135_v1120720_155254_6927", "etorassa_HIG-Summer12-00803_batch135_v1120720_155404_8501", "spinoso_EXO-Summer11-01441_R1571_B106_12_LHE_120721_155427_4296", "spinoso_EXO-Summer12-01618_R1523_B126_05_LHE_120713_163333_2235", "etorassa_BPH-Summer12-00042_batch135_v1__120720_155226_1560", "casarsa_HIG-Summer12-00884_144_v1120723_112448_4744", "spinoso_HIG-Summer11-01178_R1640_B111_06_LHE_120808_100223_7580", "spinoso_EXO-Summer11-01440_R1571_B106_11_LHE_120721_155425_5366", "etorassa_BPH-Summer12-00063_batch135_v1120720_155343_4980", "etorassa_BPH-Summer12-00048_batch135_v1__120720_155248_3786", "etorassa_BPH-Summer12-00041_batch135_v1120720_155222_2812", "spinoso_EXO-Summer12-01619_R1524_B127_01_LHE_120713_163756_8438", "spinoso_EXO-Summer12-01706_R1608_B145_07_LHE_120725_201337_9332", "casarsa_HIG-Summer12-00891_144_v1120723_112521_2384", "spinoso_HIG-Summer11-01173_R1640_B111_01_LHE_120808_100211_7402", "spinoso_EXO-Summer11-01450_R1571_B106_21_LHE_120721_155446_1958", "spinoso_TOP-Summer12_FS52-00138_R1599_B3_FSIM_52_LHE_120808_165325_1955", "etorassa_HIG-Summer11-01094_batch103_v1__120706_115955_6799", "etorassa_BPH-Summer12-00051_batch135_v1120720_155259_577", "spinoso_EWK-Summer12-00099_R1500_B124_01_LHE_120720_144144_9041", "etorassa_BPH-Summer12-00045_batch135_v1__120720_155236_9475"]', 'descending': 'true', 'include_docs': 'true', 'resources': '{"T2_DE_RWTH": 100.0, "T2_EE_Estonia": 100.0, "T2_PT_NCG_Lisbon": 80.0, "T2_UA_KIPT": 50.0, "T2_UK_SGrid_RALPP": 200.0}', 'teams': '["dataops"]'} result: None