Job log management improvement ideas

[HappyRay]

Here are my early rough ideas:

Jobs write logs to a log file specified by AZ runtime. Jobs can decide the format and content of the logs. e.g. they can choose log4j, log4j2 etc and their own log4j configs.

AZ web server fetches the logs directly from executors.

Separate AZ logs from job logs.

Separate job logs management/cleanup from the executor directory management/cleanup.

Limit the size of the logs shown in the log UI.

Provide a way for users to download the complete logs.

[kunkun-tang]

The idea sounds good. Do you also want to render users the full log config flexibility? Some log4j configs example is

• enable log level specified by the user;
• determine which class need to be logged, and which class doesn't

I think we should provide these support as well. AZ may define a default log4j config, and jobs are able to override part of it.

[HappyRay]

Do you also want to render users the full log config flexibility

Yes. It's part of what I meant by "Jobs can decide the format and content of the logs"

AZ may define a default log4j config, and jobs are able to override part of it.

I think it makes more sense to make this part job type specific. This is not so different from letting each process/application decide about their logging configs. In this context, each job process corresponds to a job type. This part needs to be designed. This is related to the job type re-design we talked about.