search

AgnostiqHQ / covalent-slurm-plugin

Executor plugin interfacing Covalent with Slurm
https://covalent.xyz
Apache License 2.0
27 stars 6 forks source link

Accurate time logging for queued jobs #50

Closed Andrew-S-Rosen closed 1 year ago

Andrew-S-Rosen commented 1 year ago

What should we add?

Once the job is submitted to the SLURM queue, the job is marked as "running" and the runtime starts ticking on the UI. However, this does not reflect the actual state of the job since it might be queuing. It might be difficult to modify the "running" state on the UI since there's no "queued" state (that I'm aware of), but an alternative might be to keep it as "running" but have the runtime be 0 s until it's detect that it changed state on the cluster, at which point the time can start.

Describe alternatives you've considered.

No response

santoshkumarradha commented 1 year ago

Hey @arosen93 thanks for this, indeed. We have this as a backlog - https://github.com/AgnostiqHQ/covalent-slurm-plugin/issues/39

We have a revamped status epic to allow for executor dependent status for electrons. We will be getting to this soon, we have made few more design choices that should make this easier to implement now than when this old issue was created (hence the wait).

I also like the idea of having runtime as a separate entity, but I think combining with this issues https://github.com/AgnostiqHQ/covalent-slurm-plugin/issues/39 , it can be more elegant to have the status object have a time stamp associated so that user can directly query the time from status change if they want to calculate it. Please feel free to modify this current issue to either have time stamp or close it as it can be combined with the old issue.

Andrew-S-Rosen commented 1 year ago

Ah, thanks for pointing me to that one! Looks like I missed in the hustle and bustle of debugging! I'll close the issue here since it's already been noted.