[FEATURE] A more thorough data cleaning mechanism

[rickyma]

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Search before asking

• [X] I have searched in the issues and found no similar issues.

Describe the feature

We found that after the server cluster is restarted, the app directories on HDFS may not be cleaned up, which can lead to long-term occupation of disk space. We need a more thorough data cleaning mechanism to make sure data is always cleaned up, without leaks.

Motivation

No response

Describe the solution

No response

Additional context

No response

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

[rickyma]

@zuston @dingshun3016 @yl09099 @xianjingfeng Have you ever encountered this issue in prod?