Pickle file paths are not handled appropriately when a `chdir` call is made

[Andrew-S-Rosen]

Environment

• Covalent version: 0.234.0rc0
• Covalent-Slurm plugin version: 98e835f29a3bf6b0fa1329051369d461cd81dfe8
• Python version: 3.10.13
• Operating system: Linux

What is happening?

When a @ct.electron changes the working directory, Covalent crashes because it can no longer find the results pkl file. See #94 for the somewhat cryptic error message.

How can we reproduce the issue?

import covalent as ct

executor = ct.executor.SlurmExecutor(
    username="rosen",
    address="perlmutter-p1.nersc.gov",
    ssh_key_file="/home/rosen/.ssh/nersc",
    cert_file="/home/rosen/.ssh/nersc-cert.pub",
    conda_env="covalent",
    options={
        "nodes": 1,
        "qos": "debug",
        "constraint": "cpu",
        "account": "matgen",
        "job-name": "test",
        "time": "00:10:00",
    },
    remote_workdir="/pscratch/sd/r/rosen/test",
    create_unique_workdir=True,
    cleanup=False,
)

@ct.lattice(executor=executor)
@ct.electron
def workflow():
    import os

    os.chdir("../")
    return os.getcwd()

ct.dispatch(workflow)()

What should happen?

No crash.

Any suggestions?

Always, always, always have all files generated/parsed by Covalent be internally represented as absolute file paths, never relative file paths. You can basically copy what I've done in the covalent-hpc-plugin here and elsewhere in terms of file path handling.