[Core feature] Ability to Capture Pipe stderr and stdout

[rkrishnasanka]

Motivation: Why do you think this is important?

Currently, I'm running algorithms that take over one week to run on the flight cluster; one of the best ways to keep track of the current status of the solver is to check the stdout.

So the ability to pipe the stdout or the stderr to a host:port would be the best way to go about it.

Goal: What should the final outcome look like, ideally?

Being able to use a syntax like:

def stdout_pipe_callback(data: str):
   # send data to "https://myfavoritehost:1234"
   ...

def stderr_pipe_callback(data: str):
   # send data to "https://myotherfavoritehost:1234"
   ...

@stream_task(stdout_pipe=stdout_pipe_callback, stderr_pipe=stderr_pipe_callback)
def do_some_work(input:InputType) -> OutputType:
 .....
 return results

This would allow the user, to define custom callback methods to stream back the stdout.

Describe alternatives you've considered

I've considered using the shell task, but that won't allow me to "stream" the outputs until the end of the process. But I need help defining python code to use.

Another alternative I'm considering is to figure out a way to modify the docker container to monitor all the python stdout. I haven't had any luck with that.

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• [X] Yes

Have you read the Code of Conduct?

• [X] Yes

[welcome[bot]]

Thank you for opening your first issue here! 🛠

[eapolinario]

@rkrishnasanka , IIUC, you're asking for a way to ship logs from the container directly to some logging backend. The recommended way of doing this is by either running a logging agent (e.g. fluentd) on the nodes and other options (as described here). Note that exposing a logging backend from within the container is not covered.

Configuring the shipment of logs at the task-level does not seem like the right level of abstraction (since you can achieve the same goal by configuring a streaming sidecar container, for example). Let me know what you think.

[pingsutw]

The recommended way of doing this is by either running a logging agent (e.g. fluentd) on the nodes and other options (as described here). Note that exposing a logging backend from within the container is not covered.

We can add this to our documentation.

[rkrishnasanka]

I know I haven't come back to this in a long time, the main need for me to do this was to have some kind mechanism to stream the stdout / stderr real time. Better resources on how to setup a streaming sidecar container would also be helpful. My main concern was that some logging that isn't task level will also get captured and pollute the stream. Looking forward to seeing updates on this. I'm resuming this project in the next few months so I'd like to be able to take advantage of this.