AgnostiqHQ / covalent-slurm-plugin

Executor plugin interfacing Covalent with Slurm
https://covalent.xyz
Apache License 2.0
27 stars 6 forks source link

Setting the executor in a `@ct.lattice` decorator does not use the right configuration parameters #74

Closed Andrew-S-Rosen closed 1 year ago

Andrew-S-Rosen commented 1 year ago

Environment

What is happening?

Passing in a ct.executor.SlurmExecutor definition into a @ct.lattice decorator's executor kwarg does not work properly. I am getting the dreaded "username is a required parameter in the Slurm plugin" error message even though the username is clearly shown in the UI. Unlike in #70, this is not a sublattice but rather a very simple workflow. It's clear that the covalent.conf file is being read whenever the executor is passed to the Lattice object. Passing it to the Electron object works as expected.

How can we reproduce the issue?

Let's take the example from the docs:

executor = ct.executor.SlurmExecutor(
    username="myname",
    address="test",
    ssh_key_file="/path/to/my/file",
    remote_workdir="/scratch/user/experiment1",
    options={
        "qos": "regular",
        "time": "01:30:00",
        "nodes": 1,
        "constraint": "gpu",
    },
    prerun_commands=[
        "module load package/1.2.3",
        "srun --ntasks-per-node 1 dcgmi profile --pause",
    ],
    srun_options={"n": 4, "c": 8, "cpu-bind": "cores", "G": 4, "gpu-bind": "single:1"},
    srun_append="nsys profile --stats=true -t cuda --gpu-metrics-device=all",
    postrun_commands=[
        "srun --ntasks-per-node 1 dcgmi profile --resume",
    ],
)

@ct.electron # (executor=executor) works fine here!
def my_custom_task(x, y):
    return x + y

@ct.lattice(executor=executor)
def workflow(x, y):
    return my_custom_task(x, y)

dispatch_id = ct.dispatch(workflow)(1, 2)

The UI will show "username is a required parameter in the Slurm plugin" even though the username is provided. In this case, I am starting from a default Covalent configuration file, which has no username by default.

Note: Using @ct.electron(executor=executor) instead of @ct.lattice(executor=executor) works fine.

What should happen?

The username should be detected.

Any suggestions?

Andrew-S-Rosen commented 1 year ago

Closed in https://github.com/AgnostiqHQ/covalent/pull/1736