search

nextflow-io / nf-quilt

Quilt plugin for Nextflow
Apache License 2.0
4 stars 2 forks source link

readme

nf-quilt

ARCHIVED: As of version 0.3.2, the nf-quilt plugin is now maintained by Quilt at quiltdata/nf-quilt.

Nextflow plugin for interacting with Quilt packages.

nf-quilt currently allows you to publish the outputs of a workflow run as a Quilt package. WHen you launch a pipeline with the nf-quilt plugin, it will publish a Quilt package upon workflow completion that contains output files published to S3.

Getting Started

To use the nf-quilt plugin, you need Nextflow 22.04 (or later) and Python 3.7 (or later).

Install the quilt-cli Python package:

pip3 install git+https://github.com/nextflow-io/nf-quilt.git#subdirectory=quilt-cli

Add the following snippet to your nextflow.config to enable the plugin:

plugins {
    id 'nf-quilt'
}

Configure the plugin with the quilt config scope in your nextflow.config. At a minimum, you should specify the package name and registry. You can also specify a list of paths to include in the Quilt package; by default, the plugin will include all output files that were published to S3.

TIP: It is recommended that you use publishDir to select outputs for the Quilt package, rather than quilt.paths, so that the Quilt package matches the actual workflow outputs.

Here's an example based on nf-core/rnaseq:

quilt {
  packageName = 'genomes/yeast'
  registry = 's3://seqera-quilt'
  message = 'My commit message'
  meta = [pipeline: 'nf-core/rnaseq']
  force = false
}

Finally, run your Nextflow pipeline with your config file. You do not need to modify your pipeline script in order to use the nf-quilt plugin. As long as your pipeline publishes the desired output files to S3, the plugin will automatically publish a Quilt package based on your configuration settings.

Reference

The plugin exposes a new quilt config scope which supports the following options:

Config option Description
quilt.packageName Name of package, in the USER/PKG format
quilt.registry Registry where to create the new package
quilt.message The commit message for the new package
quilt.meta Package-level metadata in the form of key-value pairs
quilt.force Skip the parent top hash check and create a new revision even if your local state is behind the remote registry
quilt.paths List of published files (can be path or glob) to include in the package

Tower

To use nf-quilt with Nextflow Tower, add the following lines to your pre-run script:

yum install python3-pip -y
yum install git -y
pip3 install git+https://github.com/nextflow-io/nf-quilt.git#subdirectory=quilt-cli

Development

Refer to the nf-hello README for instructions on how to build, test, and publish Nextflow plugins.