[BUG] `viash test` for nextflow workflows

What happened?

Running viash test on Nextflow workflows still doesn't work. There are two main issues:

[ ] It's not able to detect its dependencies:

viash test src/workflows/run_benchmark/config.vsh.yaml
Fetching repo for https://github.com/openproblems-bio/openproblems-v2.git
viash: Could not find dependencies 'control_methods/zeros', 'control_methods/sample', 'control_methods/ground_truth', 'control_methods/mean_outcome', 'control_methods/mean_across_celltypes', 'control_methods/mean_across_compounds', 'methods/nn_retraining_with_pseudolabels', 'methods/scape', 'methods/jn_ap_op2', 'methods/lgc_ensemble', 'methods/transformer_ensemble', 'methods/pyboost', 'metrics/mean_rowwise_error', 'metrics/mean_rowwise_correlation'

[ ] Testing nextflow_script's doesn't work

Steps to reproduce

Run viash test on any nextflow_script (with dependencies)

Expected behavior

I expect viash test to work

Relevant log output

No response

Version

Viash 0.9.0-RC6

Possible solution

No response

Confirmation

[X] I have searched the existing issues to make sure this is not a duplicate.
[X] I have provided clear and concise information about the bug.

Additional context

No response

@Grifs I can take a look at getting viash test of a nextflow_script to work. Do you think you could take a look at getting the dependencies to be detected?

The required functionality is 100% that of the namespace commands. After discussion we came to the conclusion that to have this functionality, it can be split up in a few chunks that are less all-encompassing. Some of these have been mentioned before as nice to have things so actually implementing it could make sense:

Add the option in namespace commands to pass a single config, e.g. viash ns test src/foo/config.vsh.yaml. The functionality should then be just like passing a -q name filter, but then match whatever component is in the config path.
Add a verbosity option in namespace commands
- Quiet: no extra output from components, just show the overview
- Errors: only show error output of a failed component
- All on Error: show all output in case a component fails
- All: show all output of all components (owie!)
- Auto: if only one component is being built, show "All", otherwise show "Errors"
Add the option to build to the output structure as with non-namespace components, TBC. Maybe just make it an option that basically sets --target output --flatten, but throws an error if there is more than one component? What to do with dependencies? here be dragons

The required functionality is 100% that of the namespace commands.

I think that the different between viash build|test and viash ns build|test happens at a few different levels. When compared to viash build|test, viash ns build|test:

Builds multiple components and runners instead of just one
Provides no interactive logs about the build procedure, only when something goes wrong will it show output at the end
Is able to resolve dependencies (because it's aware of the other components in the same project)

IMO item 1 and 2 is related to user experience, but item 3 is one of the two technical things we need to fix to get viash test src/myworkflow/config.vsh.yaml to work.

Overall proposed approach

The solution that would be the easiest fix for now, is to allow viash build and viast test to resolve dependencies.

However, I agree that it's a great time to think about the user experience of the commands in general:

Add the option in namespace commands to pass a single config, e.g. viash ns test src/foo/config.vsh.yaml

I understand your point that viash build|test and viash ns build|test can be made more similar to get things to work. What we discussed a few days ago was not getting viash ns test config.vsh.yaml to work, but instead let viash test (i.e. when no config is provided) test all components in the current project. Same same but different, but I think:

viash test: test all components
viash test src/workflow/config.vsh.yaml: test one component
viash build: build all components
viash build src/workflow/config.vsh.yaml: build one component

is more practical than:

viash ns test: test all components
viash ns test src/workflow/config.vsh.yaml: test one component
viash ns build: build all components
viash ns build src/workflow/config.vsh.yaml: build one component

Verbosity

Add a verbosity option in namespace commands

Quiet: no extra output from components, just show the overview

Errors: only show error output of a failed component

All on Error: show all output in case a component fails

All: show all output of all components (owie!)

Auto: if only one component is being built, show "All", otherwise show "Errors"

Mostly agreed! I think the current behaviour of viash ns build|test is to show the overview in STDOUT but show errors after something has failed in STDERR. The behaviour of viash build|test is to show all output on STDOUT interactively and show an error message when something has failed in STDERR.

	ns overview to STDOUT	build/test logs to STDOUT	build/test logs to STDERR	overall status message
viash build/test	/	interactive	/	on exit
viash ns build/test	interactive	/	on fail	on exit

Is this a good summary of what the current behaviour is?

I wonder which of the proposed verbosity options sets which values in this table.

`--output` parameter

Add the option to build to the output structure as with non-namespace components, TBC.

Agreed. If we move viash ns build/test to viash build/test, we can:

If --output is omitted, we can build component(s) to its usual target/<runner>/<namespace>/<name> location
If --output out is defined and the positional config.vsh.yaml, we can also just output the component to that directory (what about components with dependencies? :grimacing: error message that this is not supported? or do we output the dependencies in something like out/dependencies/?)
If --output out is defined and no positional config.vsh.yaml, we could build it as if using viash ns build --target out/ --flatten.
- However, when using viash ns build --target out/ --flatten it bothers me that resource paths might clash. This is unrelated to the issue, but could be something we solve at the same time. We could:
- Move resources to out/resources/<namespace>/<component> to get them to work again
- Throw an error message or at least warning message if there are output filename clashes
- Disallow --output when building multiple components with viash build

`--runner` parameter

Another different between viash build and viash ns build is that the former will build just one runner (the first one), while viash ns build build all of them. (The same is also true for viash test and viash ns test, though testing is not allowed in nextflow runners so only the executable is tested).

When running viash build/test config.vsh.yaml, should we build / test all runners?

The solution that would be the easiest fix for now, is to allow viash build and viast test to resolve dependencies.

Note that viash build|test already support resolving dependencies, the discussion started because local dependencies are not supported, right? So in that case, we should be careful with the flattening of local dependencies into a single output folder. If that is an absolute must, then they should probably be placed in the dependencies folder somehow.

viash-io / viash