job management API - job I/O questions

[danielskatz]

It would be good to document assumptions about job I/O. Files are mentioned, so I assume jobs can read and write to local files. Can the files be remote as well (http, globus, ftp, etc.)? Can streams be worked with? What about command-line parameters to the job? What about environment variables (can they be transferred or set)? Is there an equivalent of stdout and stderr? If so, are these staged in some way?

[andre-merzky]

Hi @danielskatz - the document is indeed silent about any data staging at the moment, and limits itself to job management. We probably (and obviously) should make this clearer. We expect that data staging (and possible data management) follow in later versions of this document.

Command line parameters are covered in the JobSpecification (see the arguments field). Similar to environment variables (see environment field).

[hategan]

@danielskatz

• jobs can do what jobs can do; it's not quite for this document to get into that business; if you are referring to file staging in Layer 1, that's something else, but we are not writing that part of the spec yet.
• I don't understand "What about command-line parameters to the job?". What about them?
• I am not sure what you mean by transferring environment variables. I am assuming you are referring to inheriting them. If so, please see https://github.com/ExaWorks/job-api-spec/blob/main/specification.md#jobspecification-setenvironment and https://github.com/ExaWorks/job-api-spec/blob/main/specification.md#jobspecification-setoverrideenvironment
• Processes have a stdout and stderr and the API allows you to specify the file path where the contents should go
• Nothing is staged in Layer 0.

Maybe this the following will help: this doc describes the very basic of job submission, as you would do if you had a prompt on the head node of the machine; we call it Layer 0; the closest to it, in spirit, is DRMAA.

[danielskatz]

HI @hategan - perhaps my comments were not as clear as they could have been. My suggestion is that early in the document, there should be a subsection that briefly introduces the model of a job being used, specifically including the I/O options that are supported. I agree that some of these issues appear later in detail.

Also, when you write

Maybe this will help.

I don't what this refers to.

[hategan]

HI @hategan - perhaps my comments were not as clear as they could have been. My suggestion is that early in the document, there should be a subsection that briefly introduces the model of a job being used, specifically including the I/O options that are supported. I agree that some of these issues appear later in detail.

Indeed. I think one of the problems is that we are soliciting comments on a very early version of this API/document without clearly stating that this is a very early version. So there are many things we need to do to make this more suitable for general consumption. Some things we have planned, some remain to be determined.

Also, when you write

Maybe this will help.

I don't what this refers to.

That sentence should end with a ":" rather than a ".". I.e., what follows is what is meant to help. I'll edit.