d0choa
commented
3 years ago An idea here...
For diseases, targets and drugs, we could try to find what are the most commonly grounded diseases in the same publications a non-grounded entity is found. For the very frequent labels, I would expect the non-resolved entity should be very frequently found in conjunction with the resolved one. If this works, we could have a list of grounded targets associated with non-grounded terms that we could quickly review and suggest as disease synonyms for inclusion in EFO.
As part of the new ePMC pipeline, we are performing the entity normalisation ourselves. The algorithm produces a list of diseases that did not ground to any disease. We want to be able to maximise the value and increase the coverage of grounded entities.
Top 500 most frequently not grounded disease labels (out of 2,157,032 not grounded)
``` out =( failMatches .filter(F.col("type") == "DS") .groupBy("label") .agg(F.count(F.lit(1)).alias("count"), F.countDistinct(F.col("pmid")).alias("PMIDs"), F.first("text").alias("example")) .sort(F.col("count").desc()) ) >>> out.select("label", "count", "PMIDs").show(500, truncate = False) +------------------------------------------+------+-----+ |label |count |PMIDs| +------------------------------------------+------+-----+ |T2DM |320478|30076| |BC |252983|23685| |NAFLD |244002|19831| |T2D |194766|16253| |UC |182975|22051| |MetS |168151|10225| |AMI |151948|22281| |MDD |150986|16859| |TNBC |146192|10613| |chronic diseases |134989|78674| |HIV |134682|45979| |VTE |129526|13253| |PDAC |126471|8611 | |primary tumor |118362|57228| |IR |116427|24850| |solid tumors |113117|63479| |MCI |110272|15259| |chronic disease |108609|62902| |substance use |104169|32967| |T1D |102150|9548 | |OSCC |99086 |8609 | |SCC |93385 |21363| |addiction |92225 |34918| |psychological distress |88820 |30310| |glucose tolerance |85671 |42610| |OS |76655 |8637 | |autoimmunity |76533 |40577| |rabies |70893 |12164| |CIN |66089 |10994| |viremia |65261 |21212| |HBV infection |64822 |17207| |AR |63122 |11082| |IPV |59603 |5549 | |T1DM |59383 |7159 | |OC |59152 |6448 | |Alzheimerâs disease |57798 |30410| |inflammatory diseases |57181 |41651| |CoV-2 infection |56694 |15571| |CDI |56637 |4905 | |primary tumors |55013 |29649| |skin lesions |54090 |30563| |H. pylori infection |53808 |13187| |Alzheimera |53047 |22142| |PTC |49925 |7087 | |EOC |48314 |4589 | |CS |48290 |8834 | |VL |46764 |4903 | |CRS |46016 |7364 | |cholera |45891 |11327| |CHB |45610 |6027 | |HPV infection |45469 |12909| |neurodegenerative disorders |45280 |30848| |SAH |44736 |10667| |POAG |42728 |4951 | |EC |42652 |5949 | |PTB |41715 |4995 | |iron deficiency |41596 |15507| |Parkinsonâs disease |41557 |22164| |NCDs |41443 |6283 | |metastatic breast cancer |40870 |19756| |NETs |40835 |5887 | |chronic liver disease |40651 |24664| |renal impairment |40594 |19508| |HS |40180 |6951 | |NET |39948 |7205 | |hepatitis C |39932 |22525| |cognitive dysfunction |39404 |21796| |GD |39312 |4716 | |progressive disease |38865 |26647| |LBP |38729 |5419 | |DVT |38715 |10279| |CRPC |37875 |4886 | |brain metastases |37508 |11275| |AOM |37161 |4630 | |chronic infection |36321 |21304| |acute ischemic stroke |36085 |16306| |CIA |35794 |4571 | |NCD |35430 |5648 | |IgAN |35233 |2863 | |arterial hypertension |34395 |24116| |chronic heart failure |34304 |20370| |sexual abuse |34019 |13523| |TBI |33655 |9389 | |HFpEF |33325 |3227 | |SCI |33088 |9901 | |ILI |32339 |3244 | |bone metastases |32122 |12650| |renal dysfunction |31893 |18579| |HCV infection |31605 |11722| |hyperglycaemia |31493 |14207| |abdominal obesity |31358 |12019| |incontinence |30978 |15618| |solid tumor |29504 |19780| |systemic disease |29406 |23065| |BSE |29279 |4216 | |EVD |28992 |2414 | |impaired glucose tolerance |28590 |18236| |Parkinsona |28517 |13199| |Crohnâs disease |28492 |11708| |IE |27779 |3893 | |opportunistic infections |27760 |17947| |HH |27545 |4599 | |brain damage |27194 |18925| |anaphylaxis |27137 |9136 | |mCRC |26881 |3573 | |psychological stress |26498 |14335| |acute infection |26333 |16699| |chronic illness |26310 |16031| |inflammatory disease |25577 |20709| |EAC |25408 |3380 | |septicemia |25044 |16124| |OAB |24810 |3182 | |BCS |24772 |4970 | |EA |24151 |5088 | |FM |23921 |2932 | |angina |23800 |14855| |VAP |23785 |3966 | |substance use disorders |23563 |12630| |SCZ |23401 |1974 | |nosocomial infections |23360 |14791| |varicocele |23210 |4787 | |liver metastases |23091 |11530| |BCa |22723 |2119 | |HAT |22709 |2473 | |neuropsychiatric disorders |22415 |14103| |central obesity |22366 |9184 | |effusion |22302 |11626| |MCC |22151 |2696 | |miscarriage |21946 |9913 | |alexithymia |21884 |3350 | |systemic diseases |21614 |16597| |DKD |21526 |1549 | |SUD |21474 |3664 | |UI |21313 |2697 | |TLE |21180 |4169 | |malaria infection |20972 |7196 | |non-communicable diseases |20917 |12764| |AUD |20461 |3147 | |GBC |20331 |1612 | |hepatic encephalopathy |20316 |10162| |childhood obesity |20055 |9060 | |community-acquired pneumonia |19840 |11220| |ALF |19836 |2467 | |cardiac fibrosis |19771 |6924 | |IGT |19753 |5503 | |NE |19640 |4058 | |dyslipidaemia |19341 |9365 | |CAC |19276 |2941 | |RSV infection |19218 |4282 | |LUAD |19181 |2159 | |liver damage |19057 |12612| |cardiac dysfunction |19048 |11795| |emotional distress |19025 |10879| |MPM |19016 |2287 | |ischaemic heart disease |18896 |11451| |left ventricular dysfunction |18698 |13526| |psychiatric illness |18655 |13208| |CRF |18574 |5224 | |coronavirus disease |18489 |12164| |OB |18411 |4008 | |EH |18287 |2973 | |wound infection |18284 |11259| |neurodegenerative disorder |18226 |15034| |LSCC |18074 |1605 | |BA |17866 |2874 | |endotoxemia |17818 |8430 | |solid tumours |17754 |10154| |brain metastasis |17468 |6861 | |hemorrhagic stroke |17329 |8091 | |PsA |17118 |4022 | |HFrEF |17065 |2634 | |chronic infections |17008 |12183| |primary infection |16907 |10091| |GHD |16831 |2633 | |PKD |16827 |2856 | |liver metastasis |16591 |8674 | |delusions |16523 |8046 | |traumatic brain injury |16522 |12556| |SSI |16504 |4668 | |interstitial fibrosis |16185 |10285| |ATL |16051 |3106 | |UUO |15903 |1924 | |OLP |15849 |1588 | |plaque |15818 |9013 | |AECOPD |15805 |1485 | |NMIBC |15752 |2230 | |persistent infection |15627 |9830 | |HZ |15458 |1208 | |NMOSD |15446 |1457 | |aGVHD |15443 |2109 | |influenza A |15430 |7326 | |bone loss |15241 |8923 | |RP |15218 |4518 | |IVH |15183 |3436 | |thyroid dysfunction |15086 |7563 | |psychopathy |14991 |3206 | |bloodstream infections |14909 |8691 | |chronic illnesses |14771 |10692| |inflammatory disorders |14728 |11943| |latent infection |14705 |8218 | |cGVHD |14488 |1816 | |chronic inflammatory diseases |14484 |11261| |DENV infection |14467 |2646 | |melioidosis |14399 |2551 | |intimate partner violence |14351 |6811 | |inflammatory arthritis |14334 |7745 | |mCRPC |14327 |2277 | |mRCC |14226 |2124 | |PSC |14211 |3333 | |type 2 DM |14193 |4858 | |pulmonary infection |14185 |9000 | |FL |14159 |3117 | |ESKD |14033 |2173 | |TB infection |14020 |5469 | |influenza virus infection |14017 |6177 | |systemic infection |13987 |10190| |VaD |13897 |1986 | |substance use disorder |13862 |7723 | |hyperhomocysteinemia |13706 |6072 | |PANSS |13539 |4364 | |AHF |13518 |1943 | |steatohepatitis |13515 |6386 | |mixed infections |13442 |6953 | |anosmia |13441 |5558 | |emotional exhaustion |13438 |3603 | |invasive cancer |13437 |8332 | |DKA |13418 |2500 | |CAPD |13357 |4797 | |reflux |13301 |6927 | |allergic inflammation |13165 |7220 | |nosocomial infection |13155 |7837 | |dyskinesia |13114 |5614 | |gastrointestinal disorders |13107 |10169| |haemophilia |13078 |4812 | |PVD |13076 |3361 | |NF |13064 |2720 | |renal damage |13055 |8605 | |T. gondii infection |13032 |2950 | |age-related diseases |12923 |7794 | |mental health disorders |12898 |7339 | |CCI |12892 |3326 | |ARI |12882 |2600 | |PAF |12873 |3183 | |UCB |12848 |2236 | |nevi |12840 |4408 | |undernutrition |12834 |4971 | |KOA |12776 |966 | |somatization |12765 |5131 | |PFO |12726 |2271 | |chronically ill |12712 |8113 | |CME |12697 |3016 | |AAD |12693 |1667 | |febrile neutropenia |12663 |6947 | |chronic liver diseases |12642 |8735 | |CTEPH |12582 |1554 | |LOAD |12552 |1539 | |MIBC |12537 |1714 | |interstitial pneumonia |12523 |7149 | |iron overload |12510 |5944 | |atherosclerotic plaques |12431 |8337 | |SLI |12430 |1820 | |RC |12391 |2718 | |atherosclerotic disease |12379 |8867 | |neutrophilia |12363 |7297 | |MH |12353 |2805 | |viraemia |12270 |4169 | |wound infections |12255 |8378 | |bloodstream infection |12242 |6475 | |HIT |12161 |2442 | |yellow fever |12152 |5940 | |cancera |12110 |7347 | |acute respiratory syndrome |12078 |9269 | |BCR |12039 |1839 | |chronic inflammatory disease |12014 |10337| |chronic stroke |12008 |4343 | |acute heart failure |12008 |7159 | |brain lesions |12003 |8378 | |SZ |11848 |1990 | |BCLC |11808 |2601 | |developmental disorders |11750 |8163 | |hypercholesterolaemia |11726 |6391 | |acute gastroenteritis |11709 |6096 | |PHPT |11695 |1646 | |BPA |11673 |2547 | |scabies |11667 |3360 | |Solid Tumors |11640 |9190 | |haematoma |11530 |6678 | |sexual assault |11520 |4315 | |CCC |11491 |1685 | |FASD |11413 |1477 | |PHN |11337 |1425 | |respiratory illness |11242 |7225 | |FMS |11236 |1178 | |intraventricular hemorrhage |11178 |7023 | |carotid stenosis |11172 |5291 | |FGR |11118 |1669 | |mild cognitive impairment |11117 |8760 | |VUR |11100 |2253 | |atherosclerotic plaque |10993 |7667 | |NETosis |10991 |1684 | |malignant disease |10977 |9115 | |PVT |10959 |1458 | |ST-segment elevation myocardial infarction|10957 |6674 | |hepatocarcinoma |10817 |6408 | |oesophageal cancer |10800 |3939 | |gonorrhoea |10749 |3843 | |gastrointestinal cancer |10747 |6219 | |ILC |10741 |1813 | |POP |10717 |2090 | |impaired renal function |10715 |7581 | |ME |10671 |1655 | |haematuria |10648 |5252 | |onychomycosis |10585 |3224 | |malignant diseases |10578 |8747 | |POCD |10554 |991 | |acute hepatitis |10546 |6701 | |IAV infection |10535 |1582 | |HNC |10475 |2682 | |psychological disorders |10471 |7285 | |Mtb infection |10392 |1564 | |HA |10391 |2713 | |UA |10331 |3219 | |sCJD |10305 |864 | |ONFH |10259 |830 | |emotional stress |10211 |7091 | |hydatid cyst |10193 |4384 | |brain tumours |10191 |5079 | |cerebral hemorrhage |10160 |6206 | |stones |10123 |5482 | |VS |10115 |2108 | |IMD |10087 |1646 | |paroxysmal AF |10011 |4404 | |PIH |10009 |2095 | |mixed infection |9996 |5539 | |IPMN |9936 |2106 | |dengue infection |9916 |3151 | |metastatic renal cell carcinoma |9916 |6125 | |opportunistic infection |9883 |7236 | |atresia |9863 |5757 | |solid cancers |9851 |6675 | |acute otitis media |9836 |5339 | |tumora |9826 |5247 | |chlorosis |9708 |4692 | |IDC |9703 |3682 | |BT |9689 |2779 | |septicaemia |9680 |6559 | |CoV infection |9593 |3524 | |HPV infections |9560 |4075 | |effusions |9560 |4966 | |cervical lesions |9532 |4480 | |HGSOC |9493 |975 | |chronic hypertension |9466 |5116 | |colic |9451 |3354 | |HEV infection |9434 |1955 | |SIV infection |9380 |2315 | |addictions |9333 |5137 | |hemorrhagic shock |9316 |5025 | |T. cruzi infection |9295 |2407 | |VT |9262 |3801 | |HSV-1 infection |9257 |2637 | |maternal obesity |9203 |3404 | |persistent AF |9173 |3558 | |BPSD |9082 |1339 | |PMF |9033 |1443 | |ON |9033 |1731 | |vasculopathy |8964 |5673 | |substance misuse |8871 |4076 | |small bowel obstruction |8857 |5075 | |DPN |8810 |1626 | |CLBP |8786 |1350 | |POPF |8779 |840 | |leucopenia |8759 |5687 | |PROM |8756 |2389 | |emotional abuse |8732 |3469 | |systemic hypertension |8723 |6282 | |OHCA |8715 |2515 | |iodine deficiency |8714 |3570 | |chronic migraine |8710 |2779 | |lower respiratory tract infections |8687 |5970 | |SAP |8664 |2121 | |OPC |8576 |1638 | |iNPH |8525 |570 | |STH |8505 |973 | |Incontinence |8490 |5460 | |BPPV |8413 |1463 | |emotional problems |8411 |4788 | |myocardial ischaemia |8344 |5172 | |MPE |8300 |1115 | |memory deficits |8288 |5854 | |TNBCs |8274 |2012 | |insulitis |8243 |2691 | |Cushing's disease |8239 |3902 | |SMI |8225 |1867 | |brain abscess |8210 |4068 | |lung infection |8164 |5260 | |OPD |8132 |3048 | |P. falciparum malaria |8122 |3297 | |thalassaemia |8112 |2505 | |tuberculous meningitis |8111 |4764 | |POAF |8099 |518 | |RMS |8020 |2110 | |apathy |7997 |1991 | |SA |7990 |2254 | |snakebite |7974 |1534 | |necrotizing fasciitis |7966 |4060 | |type 1 |7930 |5805 | |CIPN |7882 |846 | |pneumococcal disease |7882 |3694 | |ocular diseases |7870 |5530 | |DCD |7869 |1247 | |GH deficiency |7860 |4143 | |maternal depression |7840 |3041 | |breast tumours |7823 |3681 | |ODD |7819 |1835 | |NCC |7815 |1378 | |microbial infections |7810 |6321 | |febrile illness |7803 |4761 | |EGC |7772 |1500 | |Huntingtonâs disease |7771 |5088 | |IOM |7771 |2335 | |UBC |7735 |1161 | |SCCs |7705 |3262 | |cytopenia |7703 |4512 | |acute infections |7687 |5952 | |peritoneal carcinomatosis |7641 |3880 | |WNV infection |7630 |1769 | |IFG |7620 |2345 | |CoV-2 |7614 |4572 | |IDA |7607 |2340 | |systemic infections |7604 |6040 | |Hepatitis C |7591 |5260 | |catatonia |7577 |2017 | |PS |7563 |2163 | |PSD |7547 |1152 | |acute malnutrition |7545 |2118 | |PNES |7461 |776 | |ocular disease |7460 |5480 | |child abuse |7448 |3554 | |neuromuscular disorders |7444 |5315 | |GA |7404 |1846 | |parasitaemia |7378 |1924 | |malarial |7369 |3908 | |M. tuberculosis infection |7367 |2828 | |noncommunicable diseases |7312 |4562 | |PTE |7307 |1455 | |biliary obstruction |7282 |4551 | |NMD |7281 |1637 | |allergen |7255 |5211 | |paroxysmal atrial fibrillation |7232 |4835 | |stress incontinence |7209 |4473 | |ischemic brain injury |7197 |4470 | |organ failure |7193 |4481 | |Internet addiction |7179 |1304 | |cytopenias |7150 |4293 | |lower respiratory tract infection |7139 |4897 | |OUD |7139 |1199 | |allergic airway inflammation |7138 |3344 | |phototoxicity |7126 |3816 | |hypercalciuria |7107 |3594 | |TSCC |7104 |641 | |microbial infection |7092 |5703 | |upper airway obstruction |7086 |4625 | |CBS |7083 |1232 | |pulmonary infections |7074 |5220 | |tropical diseases |7073 |4482 | |SBP |7069 |1396 | |P. falciparum infection |7067 |2792 | |IGD |7056 |652 | |cerebral edema |7040 |4589 | |NED |7039 |1906 | |sexual violence |7008 |2145 | |IAH |6997 |738 | |aSAH |6994 |1211 | |pancreatic fistula |6981 |2989 | |thyroid disorders |6971 |4337 | |skin lesion |6963 |4413 | |MHD |6944 |1128 | |subfertility |6943 |3891 | |estrogen deficiency |6916 |3894 | |ACM |6909 |920 | |knee OA |6894 |2420 | |brain tumour |6886 |3441 | |AGC |6823 |1147 | |DTC |6795 |2602 | |pseudarthrosis |6776 |3552 | |allergic sensitization |6758 |3241 | |IPH |6754 |976 | |CHIKV infection |6701 |1113 | |hypoxaemia |6697 |3511 | |chronic schizophrenia |6673 |4113 | |NVAF |6668 |1211 | |acute respiratory infections |6666 |4392 | |physical abuse |6663 |3499 | |halitosis |6658 |1911 | |cerebral vasospasm |6655 |3249 | |colorectal liver metastases |6652 |3434 | |PT |6650 |1711 | |obstructive sleep apnoea |6646 |3944 | |ischemia-reperfusion |6645 |5338 | |CNS disease |6645 |3810 | +------------------------------------------+------+-----+ only showing top 500 rows ```