Document behavior of hybrid workflows comprehensively

[bentsherman]

Inspired by this Slack thread. The Nextflow docs mention hybrid workflows for AWS Batch and Google Cloud. However it would be good to have a dedicated section on hybrid workloads:

• describe different possible combinations (local / HPC / cloud / K8s)
• show some diagrams
• caveats and limitations (especially a warning about egress cost)

This section might be easier to place after the website and docs overhaul.

[mribeirodantas]

I was discussing this with @abhi18av the other day. It's a topic I'm highly interested in, but not really knowledgeable about this (for now, I'm working on that 😄). Are you up for a call next week to do some tests/discussions @bentsherman @abhi18av? Feel free to suggest more people if you think that could benefit the discussion.

[bentsherman]

Sure, we can meet to talk about it further. I think you could experiment with some combinations to see what is possible (e.g. local to AWS, AWS to local), and that would be a good way for you to get some experience running pipelines.

[mribeirodantas]

I will soon get some training/sandboxes. That will be great to try this out!

[abhi18av]

Yup, this is a great feature which is not highlighted in the docs. Connecting with @mribeirodantas today to see if we can publish a good showcase for this.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[edmundmiller]

Stumbled on this issue.

I wish there was a simple "ssh" executor.

For example we have several GPU machines that can't be added to our cluster, but they're accessible on the network.

The job would run just like the local except it would use ssh username@remote_host command command_arguments to launch in a specified directory.

Might be better as a plugin 🤔

[edmundmiller]

Wondering if I could hack something out with the shell directive...

[bentsherman]

Might be possible with a custom executor in a plugin. You would also need to implement the commands to check the status and kill the task over ssh, in addition to submit

[edmundmiller]

Something like

process doRemoteThings {
    shell 'ssh', 'user@remote', '/bin/bash', '-euo', 'pipefail'

    '''
    echo "hello"
    '''
}

[bentsherman]

Like I said, that might cover the job submission but not the status check or cancellation (e.g. if the workflow is terminated)

[bentsherman]

@christopher-hakkaart not urgent but would be nice to document this consistently across aws / azure / google at some point