precisiontox / ptox-metadata-manager

The PrecisionTox Metadata Manager API
https://mm.precisiontox.org
GNU Affero General Public License v3.0
2 stars 1 forks source link
google-drive metadata-management

Precision Toxicology Metadata Manager

Build Badge Coverage Report Badge Code Quality badge Status Badge Docstring Coverage Badge Docstring Coverage Badge Mypy Badge Licence Badge API Doc Badge

The PrecisionTox Metadata Manager is a tool to help manage metadata about samples collected in the lab. It is part of the PrecisionTox Consortium and is used to ensure that metadata are correct and can be used to find data files.

Table of contents

Introduction

Video tutorial

Click to view the tool presentation video on youtube
Click to view video tutorial

Objectives

The precision toxicology metadata manager was created to help partners producing data in the PrecisionTox Consortium to create, find, validate and share metadata about samples they collect in the lab. The idea behind the tool is to operate at planning stage, before organisms are even exposed to compounds and samples are collected. These samples are meant to be shipped to a central partner who will perform material extractions for RNAseq and mass-spectrometry data acquisitions. The tool ensures that metadata do not contain any error, can be used to find physical samples in the boxes shipped to the core facility and that the experimental results can be produced in a FAIR and publishable way.

Consortium partners producing samples are invited to fill a form based on an experimental design defined by the consortium. The form generates Excel files containing metadata about sample exposition and collection divided into two sheets:

The spreadsheets are then uploaded to a shared Google Drive folder and opened for editing for when users hit the lab. The tool keeps track of the files uploaded to Google Drive, can import external files, and can validate the content of the spreadsheets. These validation steps are mandatory and must be performed before the files and boxes can be shipped.

Once marked as shipped, the file is locked and cannot be edited anymore.

Upon receiving the physical sample boxes, users from the receiving partner can then mark file as received.

The tool will generate a standardised version of the file using the ISA-JSON format, which can be imported into the ISA-tools suite, merged with metadata from metabolomics and transcriptomics, and deposited to public repositories such as EMBL-EBI MetaboLights and ArrayExpress repositories.

Finally, the sample metadata are registered in a purpose-built database and be requested through a RESTful endpoint. This allows users to search and retrieve sample metadata through both programmatic and web interfaces while providing stable, persistent and unique identifiers for each record.

Metadata pipeline for sample exposure and collection

Getting started

Requirements:

Setup:

Clone and install the dependencies in a virtual environment:

git clone https://github.com/precisiontox/ptox-metadata-manager.git
cd ptox-metadata-manager
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt # for production
pip install -r requirements-dev.txt # for development
mv ptmd/resources/.env.example ptmd/resources/.env # Create the .env file

Google Drive API

Before running the application you need to obtain a Google Drive ID and secret: head to your Google Cloud console and enable the Google Drive API and the Gmail API. Then, go to API & Services > Credentials and create a new OAuth client ID using the Desktop Application option. Once done, you will be presented with a client ID and a client secret you will want to copy.

Environment variables

Inside the ptmd/resources/ directory, rename .env.example to .env and fill in the values for your environment.

# Parameters for the Flask app
SQLALCHEMY_DATABASE_URL=sqlite:///:memory:
SQLALCHEMY_SECRET_KEY=A_SUPER_SECRET_KEY_TO_ENCODE_PASSWORDS
SITE_URL=http://localhost:5000

# Parameters for the Google Drive API
GOOGLE_DRIVE_CLIENT_ID=Your_google_api_id
GOOGLE_DRIVE_CLIENT_SECRET=Your_google_api_secret
GOOGLE_DRIVE_CREDENTIALS_FILEPATH=Absolute_path_to_google_drive_credentials_file
GOOGLE_DRIVE_SETTINGS_FILEPATH=Absolute_path_to_google_drive_settings_file

# Parameters for the default admin account
ADMIN_EMAIL=your@email.com
ADMIN_USERNAME=admin
ADMIN_PASSWORD=admin

The environment variables are divided into three categories:

Migrations

Copy the value of SQLALCHEMY_DATABASE_URL from the .env file and open the alembic.ini file. Replace the value of sqlalchemy.url with the value you copied. Then, run the following command:

alembic upgrade head

This will open a Google page in your browser requesting permissions for the application to access your Google Drive and Google Mail. It will download the Google API credentials file and generate the database once you are done.

You can now run the following command and accept the application in your browser. This is done once only during first initialization. It will download the Google API credentials file and generate the database once you are done. Finally, it will boot the flask API in your local host.

Note: The Google credentials can be generated on any machine and then copied to the server where the application is running if path in the .env file matches the path on the server.

Running the application

Once you've run the migrations scripts, you can start the application with:

python -m app

Once the API is booted go to http://localhost:5000/apidocs to see the Swagger documentation.

Production

In order to run the application in production you will need a web server and a WSGI. It has been tested with Apache/Passenger and Nginx/Gunicorn. You will also want to configure your web server so that files under the ptmd/resources/ directory are not served. Finally, you need to set the publishing status of your application status to 'In Production' in your Google Cloud Console. This will allow the application to refresh authentication tokens automatically.

Architecture

The frontend client

A NuxtJS web application accessible in development and production. It is responsible for the user interface and the communication with the API.

The backend API

It is hosted in this repository. It contains a Flask application exposing a REST API and is plugged to a relational database through SQLAlchemy. It is responsible for authentication, all functionalities logic and the persistence of (meta)-data.

Development process

The API documentation is provided by a Swagger documentation, letting users query the API through a web UI. The code is documented using docstrings and the documentation is available on readTheDocs. The application is entirely unit-tested, typehints are checked with mypy, code quality is surveyed by Codacy and styles are enforced by flake8, all as part of the continuous integration pipeline.

Components

The application source code is contained in the ptmd directory and divided as such:

Tests are contained in the tests directory and divided mirroring the application exact structure. They require no data files and no interaction with the database to be executed.

Database: Entity Relationship Diagram (ERD)

Database Entity Relationship Diagram (ERD)

Development

Migrating the app in production

If the application is already deployed in production you will want to run the following command to update the database. First installing the new dependencies:

pip install -r requirements.txt

Open the alembic.ini file and replace the value of sqlalchemy.url at line 63 with the value of SQLALCHEMY_DATABASE_URL from the .env file. Then, run the following command:

alembic upgrade head

You can now restart the web server and WSGI.

Testing

You will need the development dependencies installed to run the tests.

coverage run -m unittest discover -s tests/
coverage report -m

Verify and generate the documentation

The documentation is generated using Sphinx and hosted on ReadTheDocs. It can also be generated locally after installing the development dependencies. For Unix based systems:

docstr-coverage ptmd/ --fail-under=100
cd docs
make html

For Windows:

docstr-coverage ptmd/ --fail-under=100
.\docs\make.bat html

Verify type hints

You will need the development dependencies installed to verify the type hints.

mypy --config-file=./mypy.ini

Authors and publications: