Matgenix / jobflow-remote

jobflow-remote is a Python package to run jobflow workflows on remote resources.
https://matgenix.github.io/jobflow-remote/
Other
25 stars 11 forks source link

Add `jf runner show-config` #179

Open janosh opened 1 month ago

janosh commented 1 month ago

related to https://github.com/Matgenix/jobflow-remote/issues/155

for a runner who's been up for weeks and whose config file may have undergone multiple changes in that time, it would be great to be able to verify exactly which version of the config the runner is currently using. could we add a print config option like jf runner info --config or jf runner show-config

jf runner show-config
>>> name: project-name
workers:
  worker1:
    type: local
    ...

there could also be, say, a jf runner info --stale-config option that just prints up-to-date or stale when the runner is out of date with its config file. alternatively, that could also be made part of the jf runner info output

jf runner info
The selected project is proj1 from config file ...
# add another line here to say whether config has been modified since runner was started
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ Process                                      ┃ PID     ┃ State   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│ supervisord                                  │ 2501971 │ RUNNING │
│ runner_daemon_checkout:run_jobflow_checkout  │ 2501974 │ RUNNING │
│ runner_daemon_complete:run_jobflow_complete0 │ 2501975 │ RUNNING │
│ runner_daemon_queue:run_jobflow_queue        │ 2501976 │ RUNNING │
│ runner_daemon_transfer:run_jobflow_transfer0 │ 2501977 │ RUNNING │
└──────────────────────────────────────────────┴─────────┴─────────┘

finally, maybe a jf runner show-config --diff option would be useful that uses Python's difflib to highlight which lines changed in the current runner's config vs its config file

gpetretto commented 1 month ago

Hi @janosh, this seems indeed a good information to have, in order to clear the doubts about the status of the runner. Allowing to fetch information directly from the runner process could be feasible, but impractical. And probably not worth for this kind of information, that remains fixed theoughout the execution of the runner. I would propose that the DaemonManager also dumps/copies a version of the current configuration file in the ~/.jfremote/$PROJECT_NAME/daemon folder when the runner is started. Implementing the options that you suggest would then boil down to just read that file. Printing, determining if stale and print the diff will be straightforward. What do you think?

janosh commented 1 month ago

Printing, determining if stale and print the diff will be straightforward. What do you think?

sounds like a good way to go about this! i'm hoping Radical can submit a PR for that though #124 should probably get priority

davidwaroquiers commented 1 month ago

Somehow kind of related to #150 I would say. In principle, only one runner should be started for a given db, but currently nothing prevents from having two places (servers) in which a runner is started. Maybe jf runner info / show-config could also provide some information about a potential runner running elsewhere (taken from auxiliary collection). Regarding jf runner show-config, maybe it would be logical to have a verbose (-v or -vv or -vvv) to jf runner info, similar to jf job info for example. -v could be to show the config, while -vvv to show the config + difference.

Thanks for proposing to submit this PR @janosh when you have time (and/or the one about #124).