This repository contains the code and results for the paper Tartarus, an open-source collection of benchmarks for evaluation of a generative model.
To run the Tartarus benchmark we recommend using the provided Docker container. Optionally, we also provide instructions to building and running the benchmark locally. The following directions will walk you through setup and evaluation. You will need to have Docker installed on your machine. Once you have Docker installed, you can follow these steps:
Write the SMILES to be evaluated to a CSV file with a column header smiles
.
Pull the latest Tartarus Docker image:
docker pull johnwilles/tartarus:latest
docker run --rm -it -v ${LOCAL_PATH_TO_DATA}:/data johnwilles/tartarus:latest --mode ${BENCHMARK_MODE} --input_filename ${INPUT_FILENAME}
output.csv
. To install Tartarus locally, we recommend using the provided Conda environment definition.
git clone git@github.com:aspuru-guzik-group/Tartarus.git
conda env create -f environment.yml
conda activate tartarus
chmod 777 tartarus/data/qvina
chmod 777 tartarus/data/smina
Note: These executables are only compatible with Linux.
Detailed documentation can be found here: Tartarus Docs
Below are some examples of how to load the datasets and use the fitness functions. For more details, you can also look at example.py
.
All datasets are found in the datasets directory. The arrows indicate the goal (↑ = maximization, ↓ = minimization).
Task | Dataset name | # of smiles | Columns in file | |||||
---|---|---|---|---|---|---|---|---|
Designing OPV | hce.csv |
24,953 | PCEPCBM -SAS (↑) | PCEPCDTBT -SAS (↑) | ||||
Designing emitters | gdb13.csv |
403,947 | Singlet-triplet gap (↓) | Oscillator strength (↑) | Multi-objective (↑) | |||
Designing drugs | docking.csv |
152,296 | 1SYH (↓) | 6Y2F (↓) | 4LDE (↓) | |||
Designing chemical reaction substrates | reactivity.csv |
60,828 | Activation energy ΔE‡ (↓) | Reaction energy ΔEr (↓) | ΔE‡ + ΔEr (↓) | - ΔE‡ + ΔEr (↓) |
To use the evaluation function, load either the full xtb calculation from the pce
module, or use the surrogate model, with pretrained weights.
import pandas as pd
data = pd.read_csv('./datasets/hce.csv') # or ./dataset/unbiased_hce.csv
smiles = data['smiles'].tolist()
smi = smiles[0]
## use full xtb calculation in hce module
from tartarus import pce
dipm, gap, lumo, combined, pce_pcbm_sas, pce_pcdtbt_sas = pce.get_properties(smi)
## use pretrained surrogate model
dipm, gap, lumo, combined = pce.get_surrogate_properties(smi)
Load the objective functions from the tadf
module. All 3 fitness functions are returned for each smiles.
import pandas as pd
data = pd.read_csv('./datasets/gdb13.csv')
smiles = data['smiles'].tolist()
smi = smiles[0]
## use full xtb calculation in hce module
from tartarus import tadf
st, osc, combined = tadf.get_properties(smi)
Load the docking
module. There are separate functions for each of the proteins, as shown below.
import pandas as pd
data = pd.read_csv('./datasets/docking.csv')
smiles = data['smiles'].tolist()
smi = smiles[0]
## Design of Protein Ligands
from tartarus import docking
score_1syh = docking.get_1syh_score(smi)
score_6y2f = docking.get_6y2f_score(smi)
score_4lde = docking.get_4lde_score(smi)
Load the reactivity
module. All 4 fitness functions are returned for each smiles.
import pandas as pd
data = pd.read_csv('./datasets/reactivity.csv')
smiles = data['smiles'].tolist()
smi = smiles[0]
## calculating binding affinity for each protein
from tartarus import reactivity
Ea, Er, sum_Ea_Er, diff_Ea_Er = reactivity.get_properties(smi)
Our results for running the corresponding benchmarks can be found here:
Make a github issue 😄. Please be as clear and descriptive as possible. Please feel free to reach out in person: (akshat98[AT]stanford[DOT]edu, robert[DOT]pollice[AT]gmail[DOT]com)