cbizon
commented
2 years ago Yes, this is a question I have struggled with.
Looking at the sizes of the compendia themselves is a good way to get some idea what's going on. The big changes in ChemicalEntity and MolecularMixture are both concerning. To me, that suggests that both are not merging on something, so we have more smaller cliques than fewer larger ones. And of course, the new way may be better; it will require a bit of digging into.
The stuff in reports is meant as a partial answer to this problem.
The reports that are *done are not interesting, they are just artifacts for snakemake to know that the process has completed. Probably they should be written into another place? Maybe they can be dispensed with entirely with a bit of thought.
The _completeness files are there to check for a particular kind of error: making sure that we're not losing identifiers. At times past we have sometimes had input identifiers that didn't end up in any cliques. So these are lists of identifiers that didn't end up in cliques. Really, you want all these to just have the header, and you don't want them to increase in size. We previously have had a long running issue in disease: https://github.com/TranslatorSRI/NodeNormalization/issues/64 . It looks like that one got worse; if it's a question of the same cause then so be it but maybe something else has changed.
The most useful (?) files are the ones called e.g. reports/MolecularMixture.txt. First you get a cluster size distribution. It's often good to review the biggest cluster and see if anything can be done about them. Usually if there's not a bulge at the end of the distribution though, there's not much that can be done. The other useful thing is the prefix list. So for instance in that MM.txt, you see this:
frozenset({('UNII', 2), ('CAS', 1), ('PUBCHEM.COMPOUND', 2), ('INCHIKEY', 1)}) 1
frozenset({('MESH', 1), ('PUBCHEM.COMPOUND', 1), ('UNII', 2), ('CHEMBL.COMPOUND', 1), ('UMLS', 3), ('INCHIKEY', 1)}) 1
frozenset({('MESH', 1), ('PUBCHEM.COMPOUND', 1), ('HMDB', 1), ('UNII', 1), ('CHEBI', 1), ('CHEMBL.COMPOUND', 1), ('INCHIKEY', 1)}) 1
...
frozenset({('INCHIKEY', 1), ('PUBCHEM.COMPOUND', 1), ('CHEMBL.COMPOUND', 1)}) 87120
frozenset({('CAS', 1), ('PUBCHEM.COMPOUND', 1), ('INCHIKEY', 1)}) 196331
frozenset({('INCHIKEY', 1), ('PUBCHEM.COMPOUND', 1)}) 5863401So the most common clique/cluster consists of a pubchem.compound id and an inchikey, and there are 5.8 M like that. The next biggest group is that plus a CAS, with almost 200k, and so on down to clusters that only happen once, like some compund that has a mesh, a pubchem, an hmdb, a unii, a chebi, a chembl, and an inchikey.
So to figure out what's going on with the big % change in the compendia, I would look at the differences in these reports and see what is getting merged/split differently. With those big % changes I think they'll be easy to find? And then it's a matter of digging around and looking at examples and seeing if it's an improvement or not.
I've made half hearted attempts to compare directly two compendium runs to see what changed, but have never found anything I liked very much. Happy to try other ideas though.
gaurav
I've just completed my first run of Babel on Sterling (on a container with 500GB of memory!) using the changes in draft PR https://github.com/TranslatorSRI/Babel/pull/37. The results I've obtained (on Hatteras at
/scratch/gaurav/babel-outputs/2022apr4) has lots of differences from the 2022-01-01 run, but I haven't come up with a good way of summarizing the changes or figuring out if it's working "correctly".I've tried using diff/diffstat, but there are tons of changes, so it's not easy to see how signficant the changes are. I tried diffing some files individually, and was able to find a few patterns: for example, the polypeptide
LSM-37009insynonyms/Polypeptide.txtis referred to as CHEBI:125504 in the new run and INCHIKEY:GGLDQJNBYFODOM-RDCMKPLUSA-N in the previous run.Diffstat comparison of Jan 1 and Apr 4 Babel runs
``` compendia/AnatomicalEntity.txt |284873 compendia/BiologicalProcess.txt |55258 compendia/Cell.txt |15690 compendia/CellularComponent.txt |24855 compendia/ChemicalEntity.txt |6976071 compendia/ChemicalMixture.txt | 889 compendia/ComplexMolecularMixture.txt | 296 compendia/Disease.txt |654029 compendia/Gene.txt |77898179 ++--- compendia/GeneFamily.txt |55418 compendia/GrossAnatomicalStructure.txt |20397 compendia/MolecularActivity.txt |294143 compendia/MolecularMixture.txt |16366879 - compendia/OrganismTaxon.txt |4783919 compendia/Pathway.txt |104290 compendia/PhenotypicFeature.txt |700283 compendia/Polypeptide.txt | 753 compendia/Protein.txt |456484451 ++++++++++++++++----------------- compendia/SmallMolecule.txt |204804339 +++++++------- conflation/GeneProtein.txt |16857753 - reports/AnatomicalEntity.txt | 100 reports/BiologicalProcess.txt | 17 reports/Cell.txt | 73 reports/CellularComponent.txt | 60 reports/ChemicalEntity.txt | 1451 reports/ChemicalMixture.txt | 20 reports/ComplexMolecularMixture.txt | 23 reports/Disease.txt | 8278 reports/Gene.txt | 72 reports/GeneFamily.txt | 8 reports/GrossAnatomicalStructure.txt | 80 reports/MolecularActivity.txt | 70 reports/MolecularMixture.txt | 2310 reports/OrganismTaxon.txt | 12 reports/Pathway.txt | 8 reports/PhenotypicFeature.txt | 1175 reports/Polypeptide.txt | 30 reports/Protein.txt | 445 reports/SmallMolecule.txt | 8790 reports/disease_completeness.txt | 69 reports/process_completeness.txt | 4 synonyms/AnatomicalEntity.txt |624380 synonyms/BiologicalProcess.txt |224432 synonyms/Cell.txt |43534 synonyms/CellularComponent.txt |57426 synonyms/ChemicalEntity.txt |1428975 synonyms/ChemicalMixture.txt | 3566 synonyms/ComplexMolecularMixture.txt | 1672 synonyms/Disease.txt |2407347 synonyms/Gene.txt |1060645 synonyms/GeneFamily.txt |55418 synonyms/GrossAnatomicalStructure.txt |105021 synonyms/MolecularActivity.txt |393102 synonyms/MolecularMixture.txt |16811698 - synonyms/OrganismTaxon.txt |139483 synonyms/Pathway.txt |109508 synonyms/PhenotypicFeature.txt |1712740 synonyms/Polypeptide.txt | 3133 synonyms/Protein.txt |2871694 synonyms/SmallMolecule.txt |215593042 +++++++-------- 60 files changed, 525618409 insertions(+), 504434267 deletions(-) ```Probably the best way to compare the changes is by comparing line counts, which shows that most files are pretty similarly sized, except for
compendia/ChemicalEntity.txt(which is 1577.58% bigger),compendia/MolecularMixture.txt(58.43% bigger) andsynonyms/MolecularMixture.txt(56.51% bigger).Does anybody have suggestions for comparing/validating the new Babel output before we try to move it to the dev server? We could for instance dump all the IDs alphabetically and run a massive diff on that. Having some method to do this would help with #36 as well.