search

AgPipeline / openstack-terraform-nomad

Set up a Nomad cluster on OpenStack using Terraform
MIT License
6 stars 4 forks source link

Explore CyVerse for TERRA REF Pipeline #9

Closed Chris-Schnaufer closed 4 years ago

julianpistorius commented 4 years ago

Good progress made. See: meeting-notes/2019-09-05_helping-sateesh-gantry-pipeline.md

In the meeting Nirav recommended to let the transfer process trigger the workflow on successful completion of the transfer, and not to bother with IRODS rules. The controller of workflow should be controlling & triggering things. So transferring should also be a workflow. The controller can live on a VM, and it's not required to be high-powered. It just tells workers (when they call back) where to get the data and what to do. He recommended that we just use Atmo/Tombstone/Jetstream/UA HPC for compute nodes instead of HTCondor notes. This approach will give us the kind of scalability we need.

And on Slack I asked:

Question: Once the containers are integrated into the DE, and the CWL description is done, will the Makeflow engine:

  1. Call the DE API to run the workflow step, or
  2. Submit the container step directly to the HTCondor cluster

Nirav said:

  1. Makeflow engine will call the json (or make) written workflow definition (not CWL as it does not speak CWL). This will include the location of container (docker/signularity hub or private)
  2. You will submit workers (HPC, cloud) which will dial home to this makeflow engine and take task and perform the work (and update the status as the go along). You can arbitrarily add more workers and get scaling of tasks.

We are still investigating using VICE for the exploration 'workbench' and as development environments. See: Investigate CyVerse VICE as PEcAn dev environment #93