The standard logging for Luigi is to log to console only. This is bad when we want to store logging for CI builds, or when customers need to share logs.
Luigi already accepts Python log configuration files. We need to create this file on the fly, and pass to luigi.build, ex. luigi.build([task], workers=5, local_scheduler=True, logging_conf_file="luigi.conf")
However, the log can be configured only once. All subsequent log configs will be ignored.
The original idea to log to a symlink (#205) is not feasible, the filehandle of the first configure-log invocation will point to the log-target, not to the symlink.
The new idea is to log to $OUTPUT/jobs/logs/main.log
Acceptance Criteria
Create a log configuration file on the file for Luigi and pass to luigi.build()
Background
The standard logging for Luigi is to log to console only. This is bad when we want to store logging for CI builds, or when customers need to share logs.
Luigi already accepts Python log configuration files. We need to create this file on the fly, and pass to luigi.build, ex. luigi.build([task], workers=5, local_scheduler=True, logging_conf_file="luigi.conf")
However, the log can be configured only once. All subsequent log configs will be ignored.
The original idea to log to a symlink (#205) is not feasible, the filehandle of the first configure-log invocation will point to the log-target, not to the symlink.
The new idea is to log to $OUTPUT/jobs/logs/main.log
Acceptance Criteria