Open hategan opened 9 months ago
[removed stupid comment]
[removed stupid comment]
:)
I agree that the wording is a bit ambiguous and I think we meant that the native id will be available as soon as the job enters the queued state, since all executors in psij-python do that. Even if we decide to go with that option, better wording can't hurt.
[removed stupid comment]
:)
I agree that the wording is a bit ambiguous and I think we meant that the native id will be available as soon as the job enters the queued state, since all executors in psij-python do that. Even if we decide to go with that option, better wording can't hurt.
Well, my comment was stupid because I am used to the staging happening before the QUEUED
state, but that does not hold for PSIJ. IIRC, we assume that the batch system enacts the staging, and thus the job first gets QUEUED
, right? But then the statement (jobID only available in QUEUED
) still holds.
So yes, I agree with you I guess - a bit more careful wording in several places may be sufficient to address this.
I am used to the staging happening before the
QUEUED
state, but that does not hold for PSIJ. IIRC, we assume that the batch system enacts the staging, and thus the job first getsQUEUED
, right? But then the statement (jobID only available inQUEUED
) still holds.
Not for the local executor and possibly not for other executors that implement synthetic staging.
We had SUBMITTED
in Swift instead of QUEUED
. Slightly different meaning which allowed us to cleanly have SUBMITTED
before STAGE_IN
in all cases. But that quite clearly meant that SUBMITTED
and having a native ID were disconnected. We didn't actually expose the native ID, so it didn't matter.
So yes, I agree with you I guess - a bit more careful wording in several places may be sufficient to address this.
The addition of file staging complicates the native id setting in local-type executors. Whereas before, we could Popen a process and immediately set the QUEUED state with the native ID, with file staging we cannot, since files must be staged before the Popen. And since STAGE_IN > QUEUED, this leads to an impossibility if the staging is to be done in the same process as the one PSI/J runs in. Even now, in psij-python we cheat a bit, since we trigger the QUEUED state after Popen is called.
One possible solution would be to have a job management process that handles both the staging and execution. It would be the PID of this process that would become the native ID of the job. This adds complexity, since we now have to communicate state transitions to the main PSI/J process from the job management process.
Another possible solution, which leads to a simpler implementation but may complicate use, is to only guarantee that the native ID is set when the job is in an ACTIVE state. This would be a specification change.