GWAS SumStats Tools

You can access comprehensive documentation for using gwas-sumstat-tools at this link: GWAS SumStats Tools Documentation.

Overview:

There are four commands, read, format validate and gen_meta (gen_meta function is currently only accessible to internal GWAS catalog users.)

read is for:

• Previewing a data file: no options
• Extracting the field headers: -h
• Extracting all the metadata: -M
• Extacting specific field, value pairs from the metada: -m <field name>

format is for:

• Converting sumstats data file to the standard format - gwas-ssf. This is not guaranteed to return a valid standard file, because manadatory data fields could be missing in the input.
  • Generate a configuration file, which serves as a blueprint for the formatting options.
  • Test the configuration file on the first five rows of the input file.
  • Apply the configuration file to the entire input file and generate formatted output file

    [!NOTE] It is memory efficient and will take approx. 30s per 1 million records

gen_meta is for:

• Generate metadata for a data file: -m
  • Read metadata in from existing file: --meta-in <file>
  • Create metadata from the GWAS Catalog (internal use, requires authenticated API): -g
  • Edit/add the values to the metadata: -e with --<FIELD>=<VALUE>

validate is for:

• Validating a summary statistic file using a dynamically generated schema

Requirements

• python >= 3.9 and <3.12

Installation

Local installation with pip

$ pip3 install gwas-sumstats-tools

Run with Docker

The following Docker command is the equivalent to running gwas-ssf.

$ docker run -it -v ${PWD}:/application ebispot/gwas-sumstats-tools:latest

Just append any subcommands or arguments e.g.:

$ docker run -it -v ${PWD}:/application ebispot/gwas-sumstats-tools:latest validate

Usage

$ gwas-ssf [OPTIONS] COMMAND [ARGS]...

Options:

• --help: Show this message and exit.

Commands:

• validate: Validate a sumstats file
• format: Format a sumstats file
• gen_meta: generate meta-yaml file
• read: Read a sumstats file

gwas-ssf validate

Validate a sumstats file

Usage:

$ gwas-ssf validate [OPTIONS] FILENAME

Arguments:

• FILENAME: Input sumstats file. Must be TSV (may be gzipped) [required]

Options:

• -e, --errors-out: Output erros to a csv file, .err.csv.gz
• -z, --p-zero: Force p-values of zero to be allowable. Takes precedence over inferred value (-i)
• -m, --min-rows: Minimum rows acceptable for the file [default: 100000]
• -i, --infer-from-metadata: Infer validation options from the metadata file -meta.yaml. E.g. a populated field for analysis software makes p-values of zero allowable.
• --help: Show this message and exit.

gwas-ssf read

Read (preview) a sumstats file

Usage:

$ gwas-ssf read [OPTIONS] FILENAME

Arguments:

• FILENAME: Input sumstats file [required]

Options:

• -h, --get-header: Just return the headers of the file [default: False]
• --meta-in PATH: Specify a metadata file to read in, defaulting to -meta.yaml
• -M, --get-all-metadata: Return all metadata [default: False]
• -m, --get-metadata TEXT: Get metadata for the specified fields e.g. `-m genomeAssembly -m isHarmonised
• --help: Show this message and exit.

gwas-ssf format

Format a sumstats file and creating a new one. Add/edit metadata.

Usage:

$ gwas-ssf format [OPTIONS] FILENAME

Arguments:

• FILENAME: Input sumstats file. Must be TSV or CSV and may be gzipped [required]

Options:

• Options for reading the input file
  • -d, --delimiter Text: Specify the delimiter in the file, if not specified, we can automatically detect the delimiter as whitespace if your file is .txt, comma if your file is .csv, or tab if your file is *.tsv.gz. Otherwise, please specify the delimiter which can help to recognise the column correctly
  • -r, --remove_comments Text: Remove the lines starts with the given character
• Options for generating configuration file
  • -g, --generate_config Boolean: To generate the configuration file for the file needed to be formatted
  • --config_out Path:Specify the configure JSON output file
• Options for applying configuration file
  • -o, --ss-out PATH: Output sumstats file
  • -a, --apply_config Boolean: Apply the given configuration file to the file
  • -t, -test_config Boolean: Test the given configuration file to the first 5 rows of the file
  • --config_in Path: Specify a configure JSON file to read in
  • -f, --analysis_software Text: Specify the analysis software used for generating the summary statistics data
  • -s, --minimal2standard: Try to convert a valid, minimally formatted file to the standard format.This assumes the file at least has p_value combined with rsid in variant_id field or chromosome and base_pair_location. Validity of the new file is not guaranteed because mandatory data could be missing from the original file. [default: False]
• Options for batch applying configuration file
  • -b, --batch_apply Boolean: Apply configuration files to a batch of summary statistics files
  • --lsf Boolean:Running the batch process via submitting jobs via LSF
  • --slurm Boolean:Running the batch process via submitting job via Slurm

gwas-ssf gen_meta

Generate a meta-yaml file for the existing sumstats file OR edit the existing meta-yaml file.

Usage:

$ gwas-ssf gen_meta [OPTIONS] FILENAME

Example:

# Generate a meta-yaml file from GWAS API (-g) with customised fields (-e --file_type=pre-gwas-ssf) for GCST90278188.tsv files
$ gwas-ssf gen_meta --meta-out GCST90278188.tsv-meta.yaml -g GCST90278188.tsv -e --file_type=pre-gwas-ssf

Arguments:

• FILENAME: Input sumstats file. Must be TSV or CSV and may be gzipped [required]

Options:

• --meta-out PATH: Specify the metadata output file
• -g, --meta-gwas: Populate metadata from GWAS Catalog [default: False]
• -e, --meta-edit: Enable metadata edit mode. Then provide params to edit in the --<FIELD>=<VALUE> format e.g. --GWASID=GCST123456 to edit/add that value [default: False]
• --help: Show this message and exit.

Development

This repository uses poetry for dependency and packaging management.

To run the tests:

1. install poetry
2. git clone https://github.com/EBISPOT/gwas-sumstats-tools.git
3. cd gwas-sumstats-tools
4. python3 -m venv env
5. pip install poetry
6. poetry install
7. poetry run pytest -s

To make a change: branch from master -> PR to master -> poetry version -> git add pyproject.toml -> git commit -> git tag -> git push origin master --tags If all the tests pass, this will publish to pypi.

A simple toolkit for reading and formatting GWAS sumstats files from the GWAS Catalog. Built with:

• Petl
• Pydantic
• Typer