Malikbadmus
commented
7 months ago Describe the Bug
Users reported two (2) major issues with Model eos30gr
- Null output predictions for certain Molecules.
- Unable to Reproduce exact values obtained from the author's publications.
Steps taken to reproduce the behavior.
I ran predictions on a dataset of 196 molecules I_test.csv, I got a "Null" output for five (5) molecules eos30gr_output.csv
Causes
The reason for the Null output is here in the main.py file. The script below though not needed as it is redundant, applies a standardized procedure to the SMILES string from the input file, but it lacks exception handling for cases where molecules cannot be standardized thus leading to an StandardiseException: Multiple non-salt/solvate components error , leading to a null output predictions for those instances where the SMILES string cannot be standardized.
smiles = []
with open(input_file, "r") as f:
reader = csv.reader(f)
next(reader)
for r in reader:
smiles += [r[0]]
mols = []
for i, smi in enumerate(smiles):
mol = Chem.MolFromSmiles(smi)
mol = standardise.run(mol)
if mol is not None:
smi = Chem.MolToSmiles(mol)
mol = Chem.MolFromSmiles(smi)
mol.SetProp("MoleculeIdentifier", "id-{0}".format(i))
mols += [mol]I ran the standardized script on the input datasets I used, and the molecules that were not standardized corresponded with the "Null" output in our prediction.
Also, the Training Datasets that were used to train the Model in this script here, rows with null values in the "activity" column were filtered out, but there was no check for missing or duplicated SMILES string. Indeed, there were 840 duplicated SMILES strings present in the combined datasets used in training the model.
The Combined Datasets used to train the model were also converted in this form XLSX → SDF → CSV as shown here, since we are bypassing featurization of molecules with mol2vecI in training a new model, I believe direct conversion to CSV should have been used, the only featurization of molecules occurred at the LazyQsar stage.
Operating environment
Ubuntu 22.04.1 LTS
Additional context
From the Model implementation on Ersilia, it could be seen that the Mol2Vec featurization method that was used in the Publication model was absent in eos30gr, and both model also used different modeling techniques in training both models for bioactivity predictions. @DhanshreeA
Adhivp
GemmaTuron
Inyrkz
Took table S7 from the dataset of original paper https://doi.org/10.1021/acs.jcim.8b00769
Originally posted by @Adhivp in https://github.com/ersilia-os/ersilia/issues/1025#issuecomment-2002472626