New job_option/type SHELL

MSO4SC / cloudify-hpc-plugin

Plugin to allow Cloudify to deploy and orchestrate HPC resources

Apache License 2.0

8 stars 8 forks source link

New job_option/type SHELL #3

Closed victorsndvg closed 5 years ago

victorsndvg commented 6 years ago

I've seen that there are, at least, 2 job types, SBATCH and SRUN. I think it could be useful to have a new SHELL type for running shell commands in a login node.

Some use cases in which this SHELL type could be useful:

Remote desktop creation. We need to execute simple shell command in the visualization login node to create a new session.
Chained steps job:
- Step 1: compute (SRUN or SBATCH job)
- Step 2: move files and prepare the next step (SHELL job)
- Step 3: compute (SRUN or SBATCH job)

What do you think?

victorsndvg commented 6 years ago

Update

If this feature is finally implemented, I think it deprecates the bootstrap and revert deployment steps.

E.g:

# boostrap_job happens in the login node
boostrap_job:
        type: hpc.nodes.job
        properties:
            job_options: 
                type: 'SHELL'
                command: 'scripts/bootstrap.sh'
        relationships:
            - type: job_contained_in_hpc
              target: ft2_node

# parallel_job happens in compute nodes
parallel_job:
        type: hpc.nodes.job
        ...
        relationships:
            - type: job_contained_in_hpc
              target: ft2_node
            - type: job_depends_on
              target: bootstrap_job

# revert_job happens in the login node
revert_job:
        type: hpc.nodes.job
        properties:
            job_options: 
                type: 'SHELL'
                command: 'scripts/revert.sh'
        relationships:
            - type: job_contained_in_hpc
              target: ft2_node
            - type: job_depends_on
              target: parallel_job

to think ... :thinking:

victorsndvg commented 6 years ago

Hi @emepetres , any news on this?

emepetres commented 6 years ago

@victorsndvg not yet sorry, still working on the inputs & data management. I'll be doing some changes on the orchestrator between today and tomorrow, I'll implement the feature then :)

victorsndvg commented 6 years ago

Great, thanks @emepetres !

victorsndvg commented 6 years ago

Hi,

as I was diving a little on your code, here I come with an extended suggestion. SHELL job type could be interesting to perform shell commands inside a workflow (e.g. some actions in a login node).

But I think it could be also interesting to have the None WorkloadManager. I mean, to be able to interact with a machine with a ssh server but not a resouces manager, sending shell commands to a machine.

What do you think?