Data_curation

This Data curation tool provides a user-friendly CLI for treating raw data. It classifies the substances passed as input by filtering the SMILES and also applies a pre-processing of those structures to make them available for QSAR modelling. It can be used in Jupyter Notebook as well, including data selection funcionalities that still need to be implemented in the CLI.

In the future, we plan to implement it as a part of Flame modelling software (https://github.com/phi-grib/flame).

Installation:

This tool has been designed as a standalone application for Linux and also works for MacOS and Windows. It needs a suitable conda working environment where to be installed.

Download the repository:

git clone https://github.com/phi-grib/Data_curation.git

Go to the repository directory:

cd Data_curation/

Create the conda environment:

conda env create -f environment.yml

Then activate the environment:

conda activate datacuration

Install Data curation:

pip install -e .

Configuration

After installation is completed, you must run the configuration command to configure the directory where datacur will place the curated files. If it hasn't been configured previously the following command

datacur -c config

will suggest a default directory structure following the XDG specification in GNU/Linux.

To specify a custom path use the -d parameter to enter the root folder where the curated files will be placed:

datacur -c config -d /my/custom/path

will set up the curation repository to /my/custom/path/curation.

Once Data curation has been configured, the current setting can be displayed using again the command

datacur -c config

As a fallback, Data curation can also be configured using the following command

datacur -c config -a silent

This option sets up the curation repository within the Data curation installation directory (curate\curation). Unlike other options, this command does not ask permision to the end-user to create the directories or set up the repositories and is used internally by automatic installers and for software development.

Quickstarting

Data curation provides a command-line interface (CLI), dataset_curation.py, which allows for a fast and easy to implement curation of a file containing, at least, molecules with SMILES and an identifier (CAS, EC, name of the molecule, internal id etc...).

You can run the following commands from any terminal, in a computer where Data curation has been installed and the environment (datacuration) was activated (source activate datacuration in Linux).

First of all, we need to define an endpoint for our curated files:

datacur -c manage -a new -e myEndpoint

This creates a new entry in the curation repository. From now on, all our curation results will be stored there. In order to check the contents of the repository we can use the following command:

datacur -c manage -a list

Now the curation repository is totally configured and ready to store the outputs. Let's curate a sample file:

datacur -i sample_file.xlsx -e myEndpoint -c curate -r -a chem

This will take the input file sample_file.xlsx and store the curated data as a pickle file in the curation repository (curated_data.pkl). With -r we asked the program to remove problematic structures and store them in a separate file for further revision. With -a chem we perform only a chemical curation (i.e. only the SMILES are treated). Since we haven't specified SMILES column nor ID column, the program uses a predifined name for each, being 'structure' for SMILES and 'name' for ID. If we want to specify those columns, which is recommended, we have to type:

datacur -i sample_file.xlsx -e myEndpoint -c curate -a chem -s smiles_colname -id id_colname -r

In that case, our input is an Excel file and the code handles this internally using Pandas option read_excel(). If we want to use another accepted format, like csv or tsv and we know we have a specific separator that is not a comma nor a tab, we can also specify the separator using the -sep option:

datacur -i sample_file.csv -e myEndpoint -sep ':' -c curate -a chem -s smiles_colname -id id_colname -r

If we have a large file containing lots of columns but we only want to keep some of them, then the --metadata or -m option is available. It will generate an output only containing the most important columns for the curation plus the ones selected as metadata. Imagine that our file contains the columns meta1 and meta 2:

datacur -i sample_file.csv -e myEndpoint -c curate -a chem -s smiles_colname -id id_colname -r -m 'meta1,meta2'

Our output will be stored containig the columns id_colname, smiles_colname, structure_curated, substance_type_name, meta1 and meta2. If this option is not selected, all the columns are stored by default.

Also, there's an option to list all the output files in the endpoint directory using the following command:

datacur -c manage -e myEndpoint -a list

Finally, if we want to retrieve the curated data, we have the option download, where we can specify one of the accepted formats: tsv, csv, xlsx, sdf or json:

datacur -a download -c manage -e myEndpoint -f sdf

The output file will be stored in the local directory where the command has been executed.

HTT-like files

This special curation option is specified with -a htt. It basically detects which columns include floats and puts them into a file called x_matrix which is returned as a tsv. It performs the chemical curation only taking into account the molecule identifier and the SMILES column and it can be executed typing the following:

datacur -i sample_file_htt.xlsx -e myEndpoint -c curate -a htt -s smiles_colname -id id_colname -r

We can recover the output using the download option mentioned above.

ChEMBL download

To use this option we have to pass to the -i command a valid ChEMBL ID (i.e CHEMBL230) or a tabular file (csv, tsv, xlsx) containing a field with ChEMBL IDs. Then select the -a option chembl. This will connect with the ChEMBL API and download the associated data to the ID in a pandas dataframe and the program will curate the structures present there. If the input is a file, the dataframes will be concatenated and each register will be tagged with the corresponding ChEMBL ID.

Important note: this option works only for ChEMBL targets or proteins with enough assayed compounds. If you pass a compound or a non assayed target it will return a warning but it will continue working.

• Single ID:

  datacur -i CHEMBL230 -a chembl -e myEndpoint -c curate -r

• Input file:

  datacur -i input_file.csv -a chembl -e myEndpoint -c curate -r

We can choose to add -r or not to remove the problematic structures. To retrieve the results we just have to use the download option mentioned above.

Data curation commands

Management commands deserve further description:

Management commands

Technical details

Using Data curation

Data curation was designed to be used in different ways, using diverse interfaces. For example:

• Using the curate.py command described above
• As a Python package in a Jupyter Notebook
• As a Python package, importing the classes and the functions.

Licensing

Data curation was produced at the PharmacoInformatics lab (http://phi.upf.edu), in the framework of the eTRANSAFE project (http://etransafe.eu). eTRANSAFE has received support from IMI2 Joint Undertaking under Grant Agreement No. 777365. This Joint Undertaking receives support from the European Union’s Horizon 2020 research and innovation programme and the European Federation of Pharmaceutical Industries and Associations (EFPIA).

Copyright 2021 Eric March (eric.march@upf.edu)

Data curation is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation version 3.

Data curation is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with Data curation. If not, see http://www.gnu.org/licenses/.