search

apache / incubator-teaclave

Apache Teaclave (incubating) is an open source universal secure computing platform, making computation on privacy-sensitive data safe and simple.
https://teaclave.apache.org
Apache License 2.0
766 stars 159 forks source link

Aggregate task and platform logs #669

Closed henrysun007 closed 1 year ago

henrysun007 commented 1 year ago

Motivation & problem statement

Current implementation uses env_logger that outputs to std::io. It is convenient for development as the developer can see the logs immediately in the terminal. However, it is not friendly to the users. The users have no access to logs, which is not good for workflow depending on intermedia outputs, e.g., model tuning. Besides, the logging service is often required for audit in some standards of privacy-preserving platforms. It is also efficient for platform Ops to see the logs of all the distributed services through a single interface.

Proposed solution

The implementation has two parts: one is for task log and the other is for platform log.

Task log

A new logger implementation named teaclave_logger is added. It supports both env_logger and task_logger. After the execution service pulls a task successfully, task_logger is used. The logs will be buffered in memory and sent to storage service after the task is over or the buffer is full. The key of the log in storage is the task id with a sequence number in case the logs are sent multiple times. The task users can get the task log after they are authored through FE. This design is only for the execution service that handles tasks one by one. Concurrency would be handled when the execution service could handle tasks parallely in later design. It is out of the scope in this solution. When the execution service is not handling tasks, env_logger is used.

Platform log

A new logging service that aggregates logs is added. A potential backend is tantivy. Tantivy is a full-text search engine library. It has powerful features and can support quick search towards the logs. The env_logger in every service is replaced by platform_logger in teaclave_logger. All the platform logs except debug level ones will be sent to logging service. Only platform admin can see the logs through FE. The logs can be searched in tantivy way. The new service topology will be as below:

clients => authentication <-+       +----> storage <----+                            |     
                            |       |                   |                            |
clients => frontend ----------> management            scheduler <-- execution        |   ---> logging
                                    |                                                |
                                    +--> access_control                              |

                                                  =>      api endpoint connections
                                                  -> internal endpoint connections

The debug level logs will be output to std::io anyway.

Describe alternatives you've considered

quickwit: it uses AGPL license that is not friendly to teaclave. No other mature loggers written Rust are found. Feel free to recommend here.