Reproducibility

[mvisani]

Hi,

I stumbled upon your paper and I believe it could be very interesting for my project as I currently need a new way of handling chemical ontologies.

I am currently trying to reproduce your results and retrain your model. I encountered a few problems that could be fixed quite easily, I believe.

• [x] Specify the Python version in the dependencies. I tried it first with version 3.12.2 and failed to install the packages. It works with 3.11.8.
• [x] wandb is not included in your dependencies, so at first, it failed, telling me that the package was not found. I was able to fix this by installing it. I think adding it to the dependencies should resolve this.
• [x] Launching your pretraining command failed because it couldn't find the smiles.txt file. I had to download it manually from Zenodo.
• [x] Once this was done, I was able to launch the training of the model. However, after the first epoch, I encountered this error:

  lightning.fabric.utilities.exceptions.MisconfigurationException: `ModelCheckpoint(monitor='val_micro-f1')` could not find the monitored key in the returned metrics: ['train_loss', 'train_loss_step', 'val_loss', 'val_loss_epoch', 'train_loss_epoch', 'epoch', 'step']. HINT: Did you call `log('val_micro-f1', value)` in the `LightningModule`?

This is where I am not so sure what is wrong. Should I change the monitor metric to something else in the default_callbacks.yml file?

Thanks in advance !

Marco

[sfluegel05]

Hi Marco,

Thanks for your interest in our work and sorry for the late response.

• I added the version requirement and wandb dependency in commit c8252b5f61d1f3b308cd0d6148b7303974cc2157.
• The smiles.txt has to be downloaded from Zenodo because it is based on a PubChem file that is not publicly available anymore (PubChem only provides SMILES-lists for the last 3 month). If a dataset based on a recent PubChem release is sufficient for you, please use --data.class_path=chebai.preprocessing.datasets.pubchem.PubchemChem in your command (instead of SWJChem).
• You are correct about the monitor metric - default_callbacks.yml uses val_micro-f1 which can only be calculated on labeled data (the PubChem dataset is unlabeled). You could replace it in default_callbacks.yml directly or use pretraining_callbacks.yml instead.

I hope this works for you. If you have more questions, please don't hesitate to ask.

Best wishes, Simon

[mvisani]

Hi @sfluegel05 !

Everything works, thank you !