search

Rucal-Data-Solutions / datalakefoundation

Datalakehouse Foundation
GNU General Public License v3.0
3 stars 0 forks source link

Rewrite logging by using a framework. (Log4j) #11

Open Gateway1024 opened 8 months ago

Gateway1024 commented 8 months ago

The logging is currently using just println() as output, you can't filter on severity, nor have influence in the log destination,

Gateway1024 commented 6 months ago

The idea now is to create a system table (like the watermark table) for logging purposes

Gateway1024 commented 2 months ago

This is currently being worked on. We are going to hook into the spark logging (log4j)

The first step is to refactor all the current debug and info messages. Later, we will catch all those messages and create a custom appender that will write to a system table.