search

ncbo / ncbo_cron

Jobs that run on a regular basis in the NCBO infrastructure
Other
2 stars 6 forks source link

ontology processing stalls for a long time between "Completed caching" and "Completed processing" steps #45

Open alexskr opened 3 years ago

alexskr commented 3 years ago

ontology processing stalls for a long time between "Completed caching" and "Completed processing" steps

RAD - the time gap is 30m -1.5 hours PLANA - 1.5 hours TEST-CONTENT: 1.5 hours FENICS: 1.5 hours Also, notice the total time to complete processing of most ontologies are like 80+ minutes whether ontology has 24K classes of just 800.


02:22: "Completed caching ontology: MYFACILITYNAME (http://data.bioontology.org/ontologies/MYFACILITYNAME/submissions/3) in 0.02333623613230884 sec. 7 classes."]]
03:49: "Completed processing of http://data.bioontology.org/ontologies/MYFACILITYNAME/submissions/3 in 5279.67s"]
12:57: "Completed caching ontology: MYFACILITYNAME (http://data.bioontology.org/ontologies/MYFACILITYNAME/submissions/4) in 0.02826966578140855 sec. 7 classes."]]
14:23: "Completed processing of http://data.bioontology.org/ontologies/MYFACILITYNAME/submissions/4 in 5188.86s"]
14:27: "Completed caching ontology: INSNAME (http://data.bioontology.org/ontologies/INSNAME/submissions/2) in 0.027505069971084595 sec. 11 classes."]]
15:53: "Completed processing of http://data.bioontology.org/ontologies/INSNAME/submissions/2 in 5201.49s"]
15:53: "Completed caching ontology: INSNAME (http://data.bioontology.org/ontologies/INSNAME/submissions/1) in 0.025515652960166335 sec. 11 classes."]]
17:22: "Completed processing of http://data.bioontology.org/ontologies/INSNAME/submissions/1 in 5328.7s"]
18:04: "Completed caching ontology: CARELEX (http://data.bioontology.org/ontologies/CARELEX/submissions/45) in 0.913087101187557 sec. 322 classes."]]
19:35: "Completed processing of http://data.bioontology.org/ontologies/CARELEX/submissions/45 in 5589.75s"]
03:31: "Completed processing of http://data.bioontology.org/ontologies/UINAMES/submissions/1 in 0.46s"]
08:47: "Completed caching ontology: INSNAME (http://data.bioontology.org/ontologies/INSNAME/submissions/3) in 0.025518028065562248 sec. 12 classes."]]
10:14: "Completed processing of http://data.bioontology.org/ontologies/INSNAME/submissions/3 in 5253.75s"]
10:21: "Completed caching ontology: CC (http://data.bioontology.org/ontologies/CC/submissions/1) in 0.018692204961553216 sec. 3 classes."]]
11:49: "Completed processing of http://data.bioontology.org/ontologies/CC/submissions/1 in 5269.2s"]
11:52: "Completed caching ontology: MYFACILITYNAME (http://data.bioontology.org/ontologies/MYFACILITYNAME/submissions/6) in 0.025941963074728847 sec. 12 classes."]]
13:19: "Completed processing of http://data.bioontology.org/ontologies/MYFACILITYNAME/submissions/6 in 5278.05s"]
16:37: "Completed caching ontology: RBO (http://data.bioontology.org/ontologies/RBO/submissions/12) in 1.4996005960274488 sec. 1245 classes."]]
18:04: "Completed processing of http://data.bioontology.org/ontologies/RBO/submissions/12 in 5246.5s"]
05:22: "Completed caching ontology: INSNAME (http://data.bioontology.org/ontologies/INSNAME/submissions/4) in 0.03617303213104606 sec. 23 classes."]]
06:49: "Completed processing of http://data.bioontology.org/ontologies/INSNAME/submissions/4 in 5273.49s"]
10:52: "Completed caching ontology: VODANA-TERMS (http://data.bioontology.org/ontologies/VODANA-TERMS/submissions/1) in 0.020728365052491426 sec. 3 classes."]]
12:19: "Completed processing of http://data.bioontology.org/ontologies/VODANA-TERMS/submissions/1 in 5257.48s"]
18:02: "Completed caching ontology: BLM (http://data.bioontology.org/ontologies/BLM/submissions/55) in 1.4744217647239566 sec. 278 classes."]]
19:31: "Completed processing of http://data.bioontology.org/ontologies/BLM/submissions/55 in 5371.99s"]
19:32: "Completed caching ontology: MMO (http://data.bioontology.org/ontologies/MMO/submissions/86) in 1.2810502080246806 sec. 752 classes."]]
21:01: "Completed processing of http://data.bioontology.org/ontologies/MMO/submissions/86 in 5381.53s"]
21:07: "Completed caching ontology: DOID (http://data.bioontology.org/ontologies/DOID/submissions/605) in 38.1181580577977 sec. 17374 classes."]]
22:36: "Completed processing of http://data.bioontology.org/ontologies/DOID/submissions/605 in 5699.67s"]
22:38: "Completed caching ontology: RS (http://data.bioontology.org/ontologies/RS/submissions/179) in 14.335491483099759 sec. 4850 classes."]]
00:08: "Completed processing of http://data.bioontology.org/ontologies/RS/submissions/179 in 5500.03s"]
01:02: "Completed caching ontology: NCIT (http://data.bioontology.org/ontologies/NCIT/submissions/103) in 695.23811056884 sec. 164521 classes."]]
02:35: "Completed processing of http://data.bioontology.org/ontologies/NCIT/submissions/103 in 8815.56s"]
02:36: "Completed caching ontology: XCO (http://data.bioontology.org/ontologies/XCO/submissions/97) in 3.1357196723110974 sec. 839 classes."]]
04:09: "Completed processing of http://data.bioontology.org/ontologies/XCO/submissions/97 in 5627.59s"]
04:12: "Completed caching ontology: INSNAME (http://data.bioontology.org/ontologies/INSNAME/submissions/5) in 0.040260558016598225 sec. 23 classes."]]
05:41: "Completed processing of http://data.bioontology.org/ontologies/INSNAME/submissions/5 in 5366.56s"]
10:12: "Completed caching ontology: ZONMW-CONTENT (http://data.bioontology.org/ontologies/ZONMW-CONTENT/submissions/1) in 0.01736048376187682 sec. 1 classes."]]
11:41: "Completed processing of http://data.bioontology.org/ontologies/ZONMW-CONTENT/submissions/1 in 5357.61s"]
11:42: "Completed caching ontology: WETAXTOPICS (http://data.bioontology.org/ontologies/WETAXTOPICS/submissions/3) in 0.09120672894641757 sec. 69 classes."]]
13:11: "Completed processing of http://data.bioontology.org/ontologies/WETAXTOPICS/submissions/3 in 5383.51s"]
19:09: "Completed caching ontology: NCIT (http://data.bioontology.org/ontologies/NCIT/submissions/104) in 688.7722888449207 sec. 164521 classes."]]
20:37: "Completed processing of http://data.bioontology.org/ontologies/NCIT/submissions/104 in 8457.59s"]
00:02: "Completed caching ontology: I2SV (http://data.bioontology.org/ontologies/I2SV/submissions/2) in 0.7758974093012512 sec. 187 classes."]]
01:31: "Completed processing of http://data.bioontology.org/ontologies/I2SV/submissions/2 in 5348.05s"]
02:47: "Completed caching ontology: VODANA-METRICS (http://data.bioontology.org/ontologies/VODANA-METRICS/submissions/1) in 0.18233209801837802 sec. 12 classes."]]
04:16: "Completed processing of http://data.bioontology.org/ontologies/VODANA-METRICS/submissions/1 in 5377.52s"]
16:37: "Completed caching ontology: LAND-SURFACE (http://data.bioontology.org/ontologies/LAND-SURFACE/submissions/1) in 0.3623433727771044 sec. 359 classes."]]
18:07: "Completed processing of http://data.bioontology.org/ontologies/LAND-SURFACE/submissions/1 in 5445.19s"]
18:12: "Completed caching ontology: LANDFORM (http://data.bioontology.org/ontologies/LANDFORM/submissions/1) in 0.26802981877699494 sec. 263 classes."]]
19:42: "Completed processing of http://data.bioontology.org/ontologies/LANDFORM/submissions/1 in 5394.55s"]
20:38: "Completed caching ontology: CHEBI (http://data.bioontology.org/ontologies/CHEBI/submissions/163) in 526.1410716320388 sec. 156098 classes."]]
22:09: "Completed processing of http://data.bioontology.org/ontologies/CHEBI/submissions/163 in 8842.6s"]
22:10: "Completed caching ontology: ICO (http://data.bioontology.org/ontologies/ICO/submissions/17) in 2.4594223322346807 sec. 963 classes."]]
23:43: "Completed processing of http://data.bioontology.org/ontologies/ICO/submissions/17 in 5648.58s"]
20:42: "Completed caching ontology: PMDO (http://data.bioontology.org/ontologies/PMDO/submissions/1) in 0.48175902385264635 sec. 237 classes."]]
22:11: "Completed processing of http://data.bioontology.org/ontologies/PMDO/submissions/1 in 5385.3s"]
23:31: "Completed caching ontology: VODANA-METRICS (http://data.bioontology.org/ontologies/VODANA-METRICS/submissions/3) in 0.38972833193838596 sec. 37 classes."]]
00:58: "Completed processing of http://data.bioontology.org/ontologies/VODANA-METRICS/submissions/3 in 5232.19s"]
01:46: "Completed caching ontology: VODANA-METRICS (http://data.bioontology.org/ontologies/VODANA-METRICS/submissions/4) in 0.0848501268774271 sec. 37 classes."]]
03:14: "Completed processing of http://data.bioontology.org/ontologies/VODANA-METRICS/submissions/4 in 5298.54s"]
03:16: "Completed caching ontology: UGANDA_DISEASES (http://data.bioontology.org/ontologies/UGANDA_DISEASES/submissions/5) in 0.037807232700288296 sec. 24 classes."]]
04:44: "Completed processing of http://data.bioontology.org/ontologies/UGANDA_DISEASES/submissions/5 in 5297.43s"]
04:56: "Completed caching ontology: UGANDA_DISEASES (http://data.bioontology.org/ontologies/UGANDA_DISEASES/submissions/6) in 0.06852366402745247 sec. 57 classes."]]
06:25: "Completed processing of http://data.bioontology.org/ontologies/UGANDA_DISEASES/submissions/6 in 5318.68s"]
09:31: "Completed caching ontology: UGANDA_DISEASES (http://data.bioontology.org/ontologies/UGANDA_DISEASES/submissions/9) in 0.05485564284026623 sec. 63 classes."]]
10:59: "Completed processing of http://data.bioontology.org/ontologies/UGANDA_DISEASES/submissions/9 in 5284.12s"]
23:01: "Completed caching ontology: UGANDA_DISEASES (http://data.bioontology.org/ontologies/UGANDA_DISEASES/submissions/11) in 0.06514999130740762 sec. 81 classes."]]
00:29: "Completed processing of http://data.bioontology.org/ontologies/UGANDA_DISEASES/submissions/11 in 5309.13s"]
01:06: "Completed caching ontology: UGANDA_DISEASES (http://data.bioontology.org/ontologies/UGANDA_DISEASES/submissions/13) in 0.12419476686045527 sec. 153 classes."]]
02:35: "Completed processing of http://data.bioontology.org/ontologies/UGANDA_DISEASES/submissions/13 in 5330.67s"]
03:06: "Completed caching ontology: UGANDA_DISEASES (http://data.bioontology.org/ontologies/UGANDA_DISEASES/submissions/14) in 0.23525583185255527 sec. 306 classes."]]
04:35: "Completed processing of http://data.bioontology.org/ontologies/UGANDA_DISEASES/submissions/14 in 5320.7s"]
04:36: "Completed caching ontology: VODANA-METRICS (http://data.bioontology.org/ontologies/VODANA-METRICS/submissions/5) in 0.07257265970110893 sec. 43 classes."]]
06:04: "Completed processing of http://data.bioontology.org/ontologies/VODANA-METRICS/submissions/5 in 5310.52s"]
18:01: "Completed caching ontology: BLM (http://data.bioontology.org/ontologies/BLM/submissions/56) in 1.2196846352890134 sec. 278 classes."]]
19:30: "Completed processing of http://data.bioontology.org/ontologies/BLM/submissions/56 in 5327.67s"]
19:32: "Completed caching ontology: PW (http://data.bioontology.org/ontologies/PW/submissions/90) in 5.33063644496724 sec. 2833 classes."]]
20:59: "Completed processing of http://data.bioontology.org/ontologies/PW/submissions/90 in 5310.07s"]
17:43: "Completed caching ontology: LANDFORM (http://data.bioontology.org/ontologies/LANDFORM/submissions/2) in 0.23363111820071936 sec. 263 classes."]]
19:11: "Completed processing of http://data.bioontology.org/ontologies/LANDFORM/submissions/2 in 5326.36s"]
19:13: "Completed caching ontology: PTO (http://data.bioontology.org/ontologies/PTO/submissions/433) in 4.450036203023046 sec. 1558 classes."]]
20:41: "Completed processing of http://data.bioontology.org/ontologies/PTO/submissions/433 in 5294.97s"]
09:09: "Completed caching ontology: RAD (http://data.bioontology.org/ontologies/RAD/submissions/17) in 60.56693819491193 sec. 46761 classes."]]
10:40: "Completed processing of http://data.bioontology.org/ontologies/RAD/submissions/17 in 6439.81s"]
10:43: "Completed caching ontology: FENICS (http://data.bioontology.org/ontologies/FENICS/submissions/7) in 0.1665186770260334 sec. 151 classes."]]
12:12: "Completed processing of http://data.bioontology.org/ontologies/FENICS/submissions/7 in 5350.16s"]
13:02: "Completed caching ontology: RAD (http://data.bioontology.org/ontologies/RAD/submissions/18) in 34.78721241699532 sec. 24075 classes."]]
14:34: "Completed processing of http://data.bioontology.org/ontologies/RAD/submissions/18 in 5758.34s"]
14:38: "Completed caching ontology: ZONMW-CONTENT (http://data.bioontology.org/ontologies/ZONMW-CONTENT/submissions/3) in 0.2061581863090396 sec. 220 classes."]]
15:44: "Completed processing of http://data.bioontology.org/ontologies/ZONMW-CONTENT/submissions/3 in 3989.84s"]
15:44: "Completed caching ontology: ZONMW-CONTENT (http://data.bioontology.org/ontologies/ZONMW-CONTENT/submissions/4) in 0.481090831104666 sec. 220 classes."]]
16:00: "Completed caching ontology: RAD (http://data.bioontology.org/ontologies/RAD/submissions/19) in 35.63328354014084 sec. 24075 classes."]]
17:29: "Completed processing of http://data.bioontology.org/ontologies/RAD/submissions/19 in 5659.1s"]
17:30: "Completed caching ontology: TEST-CONTENT (http://data.bioontology.org/ontologies/TEST-CONTEN
19:03: "Completed processing of http://data.bioontology.org/ontologies/TEST-CONTENT/submissions/1 in 5608.38s"]
19:06: "Completed caching ontology: PLANA (http://data.bioontology.org/ontologies/PLANA/submissions/30) in 1.6645998200401664 sec. 863 classes."]]
20:35: "Completed processing of http://data.bioontology.org/ontologies/PLANA/submissions/30 in 5416.39s"]
08:56: "Completed caching ontology: RAD (http://data.bioontology.org/ontologies/RAD/submissions/20) in 54.121775401290506 sec. 46761 classes."]]
10:26: "Completed processing of http://data.bioontology.org/ontologies/RAD/submissions/20 in 6075.46s"]
10:35: "Completed caching ontology: RAD (http://data.bioontology.org/ontologies/RAD/submissions/21) in 37.90872448310256 sec. 24075 classes."]]
12:05: "Completed processing of http://data.bioontology.org/ontologies/RAD/submissions/21 in 5688.4s"]
13:40: "Completed processing of http://data.bioontology.org/ontologies/NEO/submissions/62 in 5398.77s"]```
mdorf commented 3 years ago

The time lapse is due to ncbo/ncbo_annotator#15.