apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
https://dolphinscheduler.apache.org/
Apache License 2.0
12.84k stars 4.62k forks source link

[Feature][Log] Different tasks should not share the same log hiding strategy #13994

Closed fuchanghai closed 1 year ago

fuchanghai commented 1 year ago

Search before asking

Description

If I want to use different rules for different tasks, the following is the answer chatGPT gave me, I'm going to try to do it.any one have any other or better ideas? @EricGao888 @rickchengx @caishunfeng @ruanwenjun @SbloodyS

我使用logback框架做日志处理,并且通过MessageConverter 对某些日志内容进行隐藏,但是我每一个日志都可能要隐藏不同的内容 我该如何做


- chatGPT

你可以通过自定义MessageConverter来实现针对不同日志的内容隐藏。

首先,你需要定义一个接口,用于隐藏日志内容:

public interface LogContentHider { String hide(String content); } 然后,你可以实现该接口,针对不同的日志内容进行隐藏。例如,对于手机号码,你可以实现如下的隐藏方式:

public class PhoneLogContentHider implements LogContentHider { @Override public String hide(String content) { return content.replaceAll("(\d{3})\d{4}(\d{4})", "$1****$2"); } } 接下来,你需要实现自定义的MessageConverter,使用LogContentHider来隐藏日志内容。例如:

public class CustomMessageConverter extends ClassicConverter { private LogContentHider logContentHider;

@Override
public void start() {
    String option = getFirstOption();
    if (option != null) {
        switch (option) {
            case "phone": //这里可以看作 sqoop ,sql ,datax 等等
                logContentHider = new PhoneLogContentHider();
                break;
            // 其他隐藏方式
        }
    }
}

@Override
public String convert(ILoggingEvent event) {
    String message = event.getFormattedMessage();
    if (logContentHider != null) {
        message = logContentHider.hide(message);
    }
    return message;
}

} 最后,在logback.xml中配置自定义的MessageConverter:

%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %custom{phone}%n

这样,你就可以在日志输出时,通过指定不同的option来隐藏不同的内容了。例如: //日志打印时我们需要加个标记 logger.info("手机号码:13812345678", "phone"); 输出的日志内容将会是:

12:34:56.789 [main] INFO com.example.MyClass - 手机号码:138



### Use case

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
github-actions[bot] commented 1 year ago

Search before asking

Description

If I want to use different rules for different tasks, the following is the answer chatGPT gave me, I'm going to try to do it.any one have any other or better ideas? @EricGao888 @rickchengx @caishunfeng @ruanwenjun @SbloodyS


You can hide content for different logs by customizing MessageConverter.

First, you need to define an interface for hiding log content:

public interface LogContentHider {
    String hide(String content);
}
Then, you can implement this interface to hide different log contents. For example, for mobile phone numbers, you can implement the following hiding methods:

public class PhoneLogContentHider implements LogContentHider {
    @Override
    public String hide(String content) {
        return content.replaceAll("(\\d{3})\\d{4}(\\d{4})", "$1****$2");
    }
}
Next, you need to implement a custom MessageConverter and use LogContentHider to hide the log content. For example:

public class CustomMessageConverter extends ClassicConverter {
    private LogContentHider logContentHider;

    @Override
    public void start() {
        String option = getFirstOption();
        if (option != null) {
            switch (option) {
                case "phone":
                    logContentHider = new PhoneLogContentHider();
                    break;
                // other hidden methods
            }
        }
    }

    @Override
    public String convert(ILoggingEvent event) {
        String message = event. getFormattedMessage();
        if (logContentHider != null) {
            message = logContentHider.hide(message);
        }
        return message;
    }
}
Finally, configure a custom MessageConverter in logback.xml:

<configuration>
    <conversionRule conversionWord="custom" converterClass="com.example.CustomMessageConverter" />
    <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %custom{phone}%n</pattern>
        </encoder>
    </appender>
    <root level="debug">
        <appender-ref ref="console" />
    </root>
</configuration>
In this way, you can hide different content by specifying different options during log output. For example:

logger.info("Mobile phone number: 13812345678", "phone");
The output log content will be:

12:34:56.789 [main] INFO com.example.MyClass - Mobile Number: 138

Use case

No response

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

github-actions[bot] commented 1 year ago

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.