Unusual Local Storage Path for Pipeline Info Files in nf-core/mag with AWS S3 Outdir

lennijusten commented 8 months ago

Description

When running the nf-core/mag pipeline with the --outdir set to an AWS S3 path, I am observing that the pipeline successfully uploads output files to the specified S3 bucket, but it also creates a local directory with pipeline configuration files. This local directory mirrors the S3 bucket path format. I'm not sure if this is the intended behavior or a bug.

Steps to Reproduce

Run the nf-core/mag pipeline using a params.json with the --outdir set to an AWS S3 path.

On launching nf-core/mag it shows this correctly specified as

Input/output options
  input                   : samplesheet.csv
  outdir                  : s3://mybucket/bioproject/nf-core-mag/

Upon completion, find the pipeline output correctly uploaded to the S3 bucket at s3://mybucket/bioproject/nf-core-mag/
Notice the creation of a local directory at /home/user/nextflow-sandbox/s3:/mybucket/bioproject/nf-core/mag/ containing pipeline information files in pipeline_info.

Expected behavior

The expectation was that the pipeline info files (if stored locally) would be in a standard local directory path. Alternatively, these files could be uploaded to the S3 bucket without storing a local copy if that is the intended behavior.

Actual Behavior

The pipeline info files are stored locally at a path that mirrors the S3 path format: /home/user/nextflow-sandbox/s3:/mybucket/.... The files in this local directory and in the S3 bucket differ slightly in content and names (see additional context).

Environment

Java Version: openjdk version "11.0.20" 2023-07-18 LTS
Nextflow Version: 23.10.0 build 5889
Operating System: Linux ip-172-31-37-92.ec2.internal 5.10.198-187.748.amzn2.x86_64
Pipeline Execution Command: nextflow run nf-core/mag -profile docker -params-file params.json

Additional context

Local directory contents:

params_2023-11-10_16-34-57.json
pipeline_report.html
pipeline_report.txt S3 bucket contents differ slightly, including files like execution_report_2023-11-10_16-31-43.html, execution_timeline_2023-11-10_16-31-43.html, etc.

Command used and terminal output

No response

Relevant files

No response

System information

No response

jfy133 commented 8 months ago

Thanks @lennijusten ! Will look into it... There are a few variables for default output directories and I think one changed recently. Maybe we forgot to update it somewhere

jfy133 commented 7 months ago

Finally had 5 minutes to look into this and I can't find anything obvious at the moment :\

jfy133 commented 7 months ago

Thanks to @FriederikeHanssen, this may be solved in next template update: https://github.com/nf-core/tools/pull/2465/files

lennijusten commented 7 months ago

Thanks @jfy133 and @FriederikeHanssen. FWIW I noticed this was also happening in the nf-core/taxprofiler pipeline.

jfy133 commented 7 months ago

Appears it will be in most pipelines, so it will be updated across all.

jfy133 commented 6 months ago

Should be fixed once https://github.com/nf-core/mag/pull/553 has been merged in

nf-core / mag