Open HyeockJinKim opened 4 hours ago
kernels
table in this case to retrieve the persistent log data..logs
vfolder approach is also error-prone to inaccessible storage volumes from specific resource groups, and .logs
vfolder should be manually created by the user.
References
Motivation
Currently, there is a known issue with log collection where excessive log size leads to connection errors when querying session logs in the web UI. This issue requires urgent attention, especially as the volume of logs can increase rapidly when using model services. While proactive measures such as implementing log rotation to remove old log files or applying pagination to limit the amount of data requested at one time can be helpful, we aim to fundamentally improve the system by collecting logging information in a location separate from the agent. This separation allows for enhanced scalability, as it reduces the load on the agent and prevents potential bottlenecks. Moreover, it enables better log management and analysis, as external logging servers can provide more robust features for storage, retrieval, and querying of logs, facilitating quicker access to critical information during troubleshooting and analysis.
Main Tasks
I want to adopt a method where collected logs are not stored in the agent but instead forwarded to an external logging server. Ideally, I would like to leverage existing open-source projects rather than implementing this from scratch.
I suggest that we focus on the following tasks:
Expected Results