search

sql-machine-learning / sqlflow

Brings SQL and AI together.
https://sqlflow.org
Apache License 2.0
5.08k stars 697 forks source link

[Discussion]Using sidecar to collective workflow step logs #2294

Open Yancey1989 opened 4 years ago

Yancey1989 commented 4 years ago

Currently, workflow steps output three kinds of logs:

  1. stdout, some detailed logs of AI jobs
  2. stderr, error logs of AI jobs.
  3. Response message which pipe to SQLFlow GUI, such as diagnostic message and AI job running phase(not implement).

In the current codebase, we mixed Response message and stdout/stderr, all theses message would be shown on the Argo UI website, but Repsonse message is the protobuf format, we don't want users see them.

So we can use Kubernetes sidecar container to collective Response message and pipe to SQLFlow GUI.

typhoonzero commented 4 years ago

Adding containers always ask for CPU, memory resources, can we just filter out the logs? e.g.