want docker logging when I'm running control script interactively

[raphaelcervantes]

I often need to run control scripts interactively. I've set up some useful log message, for example: https://github.com/axiondarkmatterexperiment/dripline-orpheus/blob/b2dcfe4888e761893d2d1c28467d9b797b361dbe/data_taking_scripts/data_logging.py#L94

Those log messages are printed on the terminal, and that's only useful as long as I have the terminal open. But I have a script running for days and I still want to see those logs. In a kubernetes system, I'd like to be able to do kubectl logs relevant_dripline_pod to keep track of the logging information.

[laroque]

So this isn't really a dripline issue, but there are several considerations. The first big question is if you really mean "interactive" like the issue says, or if you just want to see logs (that is, are you also providing input to prompts?).

Some options include:

• (you should probably do this one) You could run your script in a k8s Job object (you can configure it to try and restart or not if it fails depending on your need). Then if it finishes or crashes, you still have a way to find the old logs and (the Job controller runs a pod, same as the Deployment controller) using the same kubectl logs command you mentioned.
• If you want something even scrappier, you can create a pod with a command that looks like while true; do date;sleep 10;done (and infinite loop to keep the container alive, with a little feedback that it hasn't hung). Then you can kubectl exec into the container and you could run your control script in an actually interactive terminal (install tmux if you want to keep it running after you disconnect and to reattach later).

Two more complex solutions (that you shouldn't worry about but FYI or for someone finding this later)

• It is generally a good idea to have some sort of log aggregation system in a cluster, that way you can more easily correlate logs between different pods or go find old logs from pods that have been garbage collected.
• If you have a script that runs for a long time and you want "the user" to be able to see the progress, it is probably better to have that go through the UI in some way. That could be a pre-built dashboard on top of the log aggregator of the last item, or it could do something like what the "main cavity" did before, where you have a place in a database that you can post the status messages and render them back (rows in a SQL table is fine for short/simple strings, you could imagine something fancy on top of a no-SQL with more complex state description if it was worth the time to build such a thing.

[raphaelcervantes]

Thanks @laroque for the insight.

"If you want something even scrappier, you can create a pod with a command that looks like while true; do date;sleep 10;done (and infinite loop to keep the container alive, with a little feedback that it hasn't hung). Then you can kubectl exec into the container and you could run your control script in an actually interactive terminal (install tmux if you want to keep it running after you disconnect and to reattach later)."

This is kind of what I do already. I do a kubectl exec into a dripline-orpheus pod and run python scripts interactively this way. Ufortunately, the command outputs don't reach the kubernetes logs, so this isn't great for keep track of multi-day runs.

"(you should probably do this one) You could run your script in a k8s Job object (you can configure it to try and restart or not if it fails depending on your need). Then if it finishes or crashes, you still have a way to find the old logs and (the Job controller runs a pod, same as the Deployment controller) using the same kubectl logs command you mentioned."

This seems like the best solution. But I'm trying to be done with Orpheus this week so I won't have time to implement this. Maybe something for someone else to think about.

"If you have a script that runs for a long time and you want "the user" to be able to see the progress, it is probably better to have that go through the UI in some way. That could be a pre-built dashboard on top of the log aggregator of the last item, or it could do something like what the "main cavity" did before, where you have a place in a database that you can post the status messages and render them back (rows in a SQL table is fine for short/simple strings, you could imagine something fancy on top of a no-SQL with more complex state description if it was worth the time to build such a thing."

This is actually the easiest thing for me to implement, but I think at this point, I log enough intermediate variables that I can guess the state of the system by glancing at the grafana plots.