search

nf-core / scdownstream

A single cell transcriptomics pipeline for QC, integration and making the data presentable
https://nf-co.re/scdownstream
MIT License
30 stars 7 forks source link

docs: skeleton for guiding users towards interpretation #67

Closed smoe closed 2 months ago

smoe commented 2 months ago

Adding a pointer to https://github.com/nf-core/scdownstream/issues/2 and adding insights by @nictru on slack to the output.md. Also added a table with files I found create that seemed interesting for which a short description would be of help. Still a skeleton, would wait for some initial comments prior to (doing my best to) filling them.

github-actions[bot] commented 2 months ago

nf-core lint overall result: Passed :white_check_mark: :warning:

Posted for pipeline commit bf46ede

+| ✅ 197 tests passed       |+
!| ❗  14 tests had warnings |!
### :heavy_exclamation_mark: Test warnings: * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found: `conf/igenomes.config` * [readme](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/readme) - README contains the placeholder `zenodo.XXXXXXX`. This should be replaced with the zenodo doi (after the first release). * [pipeline_todos](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/pipeline_todos) - TODO string in `README.md`: _Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file._ * [pipeline_todos](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/pipeline_todos) - TODO string in `README.md`: _Add bibliography of tools and data used in your pipeline_ * [pipeline_todos](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/pipeline_todos) - TODO string in `nextflow.config`: _Specify your pipeline's command line flags_ * [pipeline_todos](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/pipeline_todos) - TODO string in `base.config`: _Check the defaults for all processes_ * [pipeline_todos](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/pipeline_todos) - TODO string in `base.config`: _Customise requirements for specific processes._ * [pipeline_todos](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/pipeline_todos) - TODO string in `awsfulltest.yml`: _You can customise AWS full pipeline tests as required_ * [pipeline_todos](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/pipeline_todos) - TODO string in `ci.yml`: _You can customise CI pipeline run tests as required_ * [pipeline_todos](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/pipeline_todos) - TODO string in `methods_description_template.yml`: _#Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline_ * [pipeline_todos](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/pipeline_todos) - TODO string in `main.nf`: _Optionally add in-text citation tools to this list._ * [pipeline_todos](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/pipeline_todos) - TODO string in `main.nf`: _Optionally add bibliographic entries to this list._ * [pipeline_todos](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/pipeline_todos) - TODO string in `main.nf`: _Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!_ * [pipeline_todos](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/pipeline_todos) - TODO string in `output.md`: _Write this documentation describing your workflow's output_ ### :white_check_mark: Tests passed: * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `.gitattributes` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `.gitignore` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `.nf-core.yml` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `.editorconfig` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `.prettierignore` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `.prettierrc.yml` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `CHANGELOG.md` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `CITATIONS.md` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `CODE_OF_CONDUCT.md` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `LICENSE` or `LICENSE.md` or `LICENCE` or `LICENCE.md` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `nextflow_schema.json` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `nextflow.config` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `README.md` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `.github/.dockstore.yml` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `.github/CONTRIBUTING.md` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `.github/ISSUE_TEMPLATE/bug_report.yml` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `.github/ISSUE_TEMPLATE/config.yml` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `.github/ISSUE_TEMPLATE/feature_request.yml` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `.github/PULL_REQUEST_TEMPLATE.md` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `.github/workflows/branch.yml` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `.github/workflows/ci.yml` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `.github/workflows/linting_comment.yml` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `.github/workflows/linting.yml` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `assets/email_template.html` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `assets/email_template.txt` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `assets/sendmail_template.txt` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `assets/nf-core-scdownstream_logo_light.png` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `conf/modules.config` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `conf/test.config` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `conf/test_full.config` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `docs/images/nf-core-scdownstream_logo_light.png` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `docs/images/nf-core-scdownstream_logo_dark.png` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `docs/output.md` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `docs/README.md` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `docs/README.md` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `docs/usage.md` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `main.nf` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `assets/multiqc_config.yml` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `conf/base.config` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `.github/workflows/awstest.yml` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `.github/workflows/awsfulltest.yml` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File found: `modules.json` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `.github/ISSUE_TEMPLATE/bug_report.md` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `.github/ISSUE_TEMPLATE/feature_request.md` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `.github/workflows/push_dockerhub.yml` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `.markdownlint.yml` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `.nf-core.yaml` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `.yamllint.yml` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `bin/markdown_to_html.r` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `conf/aws.config` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `docs/images/nf-core-scdownstream_logo.png` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `lib/Checks.groovy` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `lib/Completion.groovy` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `lib/NfcoreTemplate.groovy` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `lib/Utils.groovy` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `lib/Workflow.groovy` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `lib/WorkflowMain.groovy` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `lib/WorkflowScdownstream.groovy` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `parameters.settings.json` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `pipeline_template.yml` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `Singularity` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `lib/nfcore_external_java_deps.jar` * [files_exist](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_exist) - File not found check: `.travis.yml` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `manifest.name` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `manifest.nextflowVersion` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `manifest.description` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `manifest.version` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `manifest.homePage` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `timeline.enabled` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `trace.enabled` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `report.enabled` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `dag.enabled` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `process.cpus` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `process.memory` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `process.time` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `params.outdir` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `params.input` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `params.validationShowHiddenParams` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `params.validationSchemaIgnoreParams` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `manifest.mainScript` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `timeline.file` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `trace.file` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `report.file` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable found: `dag.file` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable (correctly) not found: `params.nf_required_version` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable (correctly) not found: `params.container` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable (correctly) not found: `params.singleEnd` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable (correctly) not found: `params.igenomesIgnore` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable (correctly) not found: `params.name` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable (correctly) not found: `params.enable_conda` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config ``timeline.enabled`` had correct value: ``true`` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config ``report.enabled`` had correct value: ``true`` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config ``trace.enabled`` had correct value: ``true`` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config ``dag.enabled`` had correct value: ``true`` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config ``manifest.name`` began with ``nf-core/`` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable ``manifest.homePage`` began with https://github.com/nf-core/ * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config ``dag.file`` ended with ``.html`` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config variable ``manifest.nextflowVersion`` started with >= or !>= * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config ``manifest.version`` ends in ``dev``: ``1.0dev`` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config `params.custom_config_version` is set to `master` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config `params.custom_config_base` is set to `https://raw.githubusercontent.com/nf-core/configs/master` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Lines for loading custom profiles found * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - nextflow.config contains configuration profile `test` * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config default value correct: params.ambient_removal= decontx * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config default value correct: params.doublet_detection= scrublet * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config default value correct: params.doublet_detection_threshold= 1 * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config default value correct: params.integration_methods= scvi * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config default value correct: params.clustering_resolutions= 0.5,1.0 * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config default value correct: params.celltypist_model= * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config default value correct: params.cellbender_epochs= 150 * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config default value correct: params.var_aggr_method= mean * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config default value correct: params.custom_config_version= master * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config default value correct: params.max_cpus= 16 * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config default value correct: params.max_memory= 128.GB * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config default value correct: params.max_time= 240.h * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config default value correct: params.publish_dir_mode= copy * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config default value correct: params.max_multiqc_email_size= 25.MB * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config default value correct: params.validate_params= true * [nextflow_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nextflow_config) - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/ * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `.gitattributes` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `.prettierrc.yml` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `CODE_OF_CONDUCT.md` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `LICENSE` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `.github/.dockstore.yml` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `.github/CONTRIBUTING.md` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `.github/ISSUE_TEMPLATE/bug_report.yml` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `.github/ISSUE_TEMPLATE/config.yml` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `.github/ISSUE_TEMPLATE/feature_request.yml` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `.github/PULL_REQUEST_TEMPLATE.md` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `.github/workflows/branch.yml` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `.github/workflows/linting_comment.yml` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `.github/workflows/linting.yml` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `assets/email_template.html` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `assets/email_template.txt` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `assets/sendmail_template.txt` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `assets/nf-core-scdownstream_logo_light.png` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `docs/images/nf-core-scdownstream_logo_light.png` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `docs/images/nf-core-scdownstream_logo_dark.png` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `docs/README.md` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `.gitignore` matches the template * [files_unchanged](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/files_unchanged) - `.prettierignore` matches the template * [actions_ci](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/actions_ci) - '.github/workflows/ci.yml' is triggered on expected events * [actions_ci](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/actions_ci) - '.github/workflows/ci.yml' checks minimum NF version * [actions_awstest](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/actions_awstest) - '.github/workflows/awstest.yml' is triggered correctly * [actions_awsfulltest](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/actions_awsfulltest) - `.github/workflows/awsfulltest.yml` is triggered correctly * [actions_awsfulltest](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/actions_awsfulltest) - `.github/workflows/awsfulltest.yml` does not use `-profile test` * [readme](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/readme) - README Nextflow minimum version badge matched config. Badge: `23.04.0`, Config: `23.04.0` * [pipeline_name_conventions](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/pipeline_name_conventions) - Name adheres to nf-core convention * [template_strings](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/template_strings) - Did not find any Jinja template strings (187 files) * [schema_lint](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/schema_lint) - Schema lint passed * [schema_lint](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/schema_lint) - Schema title + description lint passed * [schema_lint](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/schema_lint) - Input mimetype lint passed: 'text/csv' * [schema_params](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/schema_params) - Schema matched params returned from nextflow config * [system_exit](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/system_exit) - No `System.exit` calls found * [actions_schema_validation](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/actions_schema_validation) - Workflow validation passed: clean-up.yml * [actions_schema_validation](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/actions_schema_validation) - Workflow validation passed: linting_comment.yml * [actions_schema_validation](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/actions_schema_validation) - Workflow validation passed: linting.yml * [actions_schema_validation](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/actions_schema_validation) - Workflow validation passed: awsfulltest.yml * [actions_schema_validation](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/actions_schema_validation) - Workflow validation passed: branch.yml * [actions_schema_validation](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/actions_schema_validation) - Workflow validation passed: awstest.yml * [actions_schema_validation](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/actions_schema_validation) - Workflow validation passed: ci.yml * [actions_schema_validation](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/actions_schema_validation) - Workflow validation passed: release-announcements.yml * [actions_schema_validation](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/actions_schema_validation) - Workflow validation passed: fix-linting.yml * [actions_schema_validation](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/actions_schema_validation) - Workflow validation passed: download_pipeline.yml * [merge_markers](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/merge_markers) - No merge markers found in pipeline files * [modules_json](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_json) - Only installed modules found in `modules.json` * [multiqc_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/multiqc_config) - `assets/multiqc_config.yml` found and not ignored. * [multiqc_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/multiqc_config) - `assets/multiqc_config.yml` contains `report_section_order` * [multiqc_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/multiqc_config) - `assets/multiqc_config.yml` contains `export_plots` * [multiqc_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/multiqc_config) - `assets/multiqc_config.yml` contains `report_comment` * [multiqc_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/multiqc_config) - `assets/multiqc_config.yml` follows the ordering scheme of the minimally required plugins. * [multiqc_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/multiqc_config) - `assets/multiqc_config.yml` contains a matching 'report_comment'. * [multiqc_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/multiqc_config) - `assets/multiqc_config.yml` contains 'export_plots: true'. * [modules_structure](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_structure) - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL' * [base_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/base_config) - `conf/base.config` found and not ignored. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `conf/modules.config` found and not ignored. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `ADATA_UNIFY` found in `conf/modules.config` and Nextflow scripts. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `QC_RAW` found in `conf/modules.config` and Nextflow scripts. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `SCANPY_FILTER` found in `conf/modules.config` and Nextflow scripts. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `CELDA_DECONTX` found in `conf/modules.config` and Nextflow scripts. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `CELLBENDER_REMOVEBACKGROUND` found in `conf/modules.config` and Nextflow scripts. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `SOUPX` found in `conf/modules.config` and Nextflow scripts. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `SCVITOOLS_SOLO` found in `conf/modules.config` and Nextflow scripts. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `SCANPY_SCRUBLET` found in `conf/modules.config` and Nextflow scripts. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `DOUBLETDETECTION` found in `conf/modules.config` and Nextflow scripts. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `SCDS` found in `conf/modules.config` and Nextflow scripts. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `CELLTYPES_CELLTYPIST` found in `conf/modules.config` and Nextflow scripts. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `SCVITOOLS_SCVI` found in `conf/modules.config` and Nextflow scripts. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `SCVITOOLS_SCANVI` found in `conf/modules.config` and Nextflow scripts. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `QC_FILTERED` found in `conf/modules.config` and Nextflow scripts. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `SCANPY_NEIGHBORS` found in `conf/modules.config` and Nextflow scripts. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `SCANPY_LEIDEN` found in `conf/modules.config` and Nextflow scripts. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `SCANPY_UMAP` found in `conf/modules.config` and Nextflow scripts. * [modules_config](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/modules_config) - `MULTIQC` found in `conf/modules.config` and Nextflow scripts. * [nfcore_yml](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nfcore_yml) - Repository type in `.nf-core.yml` is valid: `pipeline` * [nfcore_yml](https://nf-co.re/tools/docs/2.14.1/pipeline_lint_tests/nfcore_yml) - nf-core version in `.nf-core.yml` is set to the latest version: `2.14.1` ### Run details * nf-core/tools version 2.14.1 * Run at `2024-07-20 16:30:22`
nictru commented 2 months ago

Hey, the rough structure of the pipeline looks like this: Various steps in the pipeline produce outputs that need to be added to .obs or obsm etc. If the pipeline inputs are really large (10s of GB), always storing the whole AnnData object with the new info and in the end extracting only the relevant info from all objects will use a lot of disk space. To prevent this, this pipeline only stores the delta in each step and finally aggregates all deltas onto a single anndata object.

So most of the files you pointed out are deltas and also included in the final output. I am not sure how useful it is for people to use them in isolation.

Also we should not only document the output but also make its structure more intuitive. Currently it is still the default of nf-core: Each output is put into a directory named like the prefix (until first underline) of the process creating it. This can and should be changed, but has so far not been relevant for functionality development. An example for doing this properly can be found here.

smoe commented 2 months ago

These were magic words of yours. Would you mind me adding them to this documentation-idea-gathering-branch? That table I came up with I would then split into "use this for CELLxGENE et al." and "secondary intermediary files the contribute to the final 'AnnData' object".

There are different apparent top-level files (./adata/merged_inner.h5ad, ./adata/merged.h5ad, ./adata/merged_outer. h5ad) which were not ultimately obvious to me how these should be interpreted. Would you have some easily accessible description for those? I would then incorporate that into the output.md file.

nictru commented 2 months ago

Yes feel free to add anything you think helps people understand the pipeline to the docs.

I would also like to refer the people to cellxgene less strongly - I mean it is a good way of visualizing it and we should definitely mention it, but we also have own reports. The quality of them will improve soon once I have more time.

For the inner and outer merged objects: Basically when concatenating the various input datasets there are two ways of handling the genes: Either performing an inner or an outer join of the genes present in all datasets - that's what you will find here. Biologists often want to see all their favourite gene in the final object, so the final result will contain the outer join. For integration, only the inner join will be used, as missing genes often lead to a clustering that's more based on the presence of absence of genes, than on actual biological differences. Because of this, the integration is performed based on the intersection. We could also use a limited number of highly variable genes here.

A bit more technical background: When concatenating the anndata objects performing an outer join, the missing values will be filled with 0s instead of NaNs. This is because 0 values use nearly no memory in the sparse representation. Filling them with NaNs will make the memory blow up.

smoe commented 2 months ago

Just another iteration.

nictru commented 2 months ago

@smoe, thanks for your work so far. I just merged #71, which does similar things to this PR. However, I am sure we can recycle some things from here to make it better understandable for less experienced users.

I will now close this issue since a basic output documentation has now been established. If you want to add something from here, feel free to reopen this PR or open a new one. Thanks again for your effort, it gave me a lot of inspiration for #71!

smoe commented 2 months ago

Sorry for being barely responsive these days, it is a bit much from too many sides :) So, thank you for proceeding, will look at my run and am confident that new issues arise :)