Discussion: Cortex.cpp Logs

[0xSage]

Open Questions:

• What logging library are we using? Stdlib or 3rd party and why?
• What log levels are supported?
• Is logging configurable by user?
• How do you handle log rotation and file size limits?
• Are logs formatted consistently? e.g. timestamp, source of issue, message
• How are we passing through upstream errors, e.g. llamacpp errors
• Where are logs stored?
• Is logging threadsafe? Are there multiple processes writing to the same logs?
• Is there a Cortex command to tail logs?

[nguyenhoangthuan99]

Log library

• we are using trantor for logging feature because:
  • It is a Drogon's dependencies, so reuse trantor will save some resources
  • trantor is non-blocking I/O cross-platform, thread safe, async library.
  • Support multi streams logger (can chose to write log to different file, stream, ... )
  • Support log file rotation and set max file size: when log file exceeds max size, will roll to new file with the same prefix: <prefix>.00000.log -> <prefix>.00001.log. Currently max log file size is 20 Mb.
  • Support all needed informations: for example 20240903 06:46:55.974670 UTC 1268127 INFO Request 1727: Inference completed - llama_engine.cc:827, it includes time, timezone, process id, log level, message, source code file and line number
  • Log levels:
    • trace
    • info
    • debug
    • warning
    • error
    • fatal

Configuration Currently, log files are stored under /path_to_cortex_cpp_binary/logs/ but need to configurable, now we set it as default constant in source code

When this Feature is done, we will read from config to decide where to store log file.

Log upstreams For upstreams like llama.cpp and tensorrt llm, we have to override log method of those and redirect to right posion as we want, this task is Done for llamacpp and tensorrt llm, refer to this issue

Issues Trantor is thread safe but it keeps the log file open during the life time of server process. With Linux and MacOS, it works perfectly, but in windows, when a process is holding a file, windows prevents other process to access it. Refer to this Issue

Future features

• [ ] Support tail latest logfile