No option to show Nextflow logs when running the test suite

[CarlinLiao]

I often find myself editing the workflow Makefiles when I'm investigating issues with spt workflow configure and nextflow run so I can see what's wrong with the processes being run by Nextflow and if the files being saved are correct.

I'd suggest that, when VERBOSE=1 during testing, the contents of the Nextflow work directories and logs are dumped to stdout before they're deleted.

[CarlinLiao]

(Something else I do is delete test_proximity_pipeline.sh and test_umap_plot_creation.sh as well as add a dummy test test_a.sh when I just want to debug the cg-gnn workflow, so make module-test-workflow runs only the one script I need.)

[jimmymathews]

You can run a single test using a direct call to make on the subdirectory with that Makefile. The tricky part is providing all the environment variables that are expected by the submake invocation.

This worked for me:

SHELL=$(realpath build/build_scripts/status_messages_only_shell.sh) \
MAKEFLAGS=--no-builtin-rules \
BUILD_SCRIPTS_LOCATION_ABSOLUTE=$(realpath build/build_scripts) \
MESSAGE='bash ${BUILD_SCRIPTS_LOCATION_ABSOLUTE}/verbose_command_wrapper.sh' \
DOCKER_ORG_NAME=nadeemlab \
DOCKER_REPO_PREFIX=spt \
TEST_LOCATION_ABSOLUTE=$(realpath test) \
TEST_LOCATION=test \
    make --no-print-directory -C build/workflow test-../../test/workflow/module_tests/no_docker_test_cggnn.sh

The problem with dumping Nextflow logs by default is that they definitely overhwelm the logs in most normal cases with unimportant messages. I think a different verbose option should be implemented for this. When nextflow errors, it does dump relevant logs already by default, but this doesn't help you with monitoring normal runs.

[jimmymathews]

Also I noticed a small error in the new standlone test, will push a fix to main.

[jimmymathews]

I think my comment stands, verbose mode is probably best not showing all the nextflow logs since they are too many.