apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.85k stars 1.77k forks source link

[Feature][Zeta] Support automatic delete of completed task log files #7716

Open hailin0 opened 4 days ago

hailin0 commented 4 days ago

Search before asking

Description

Current, zeta support creating log files for each job, #7712

But because the logging framework cannot get the job status, it cannot automatically delete completed job log files.

Therefore, we can add a scheduled task to automatically delete the completed job logs.

Requirements:

  1. Check regularly every day, use corn expression configuration, users can change the configuration, and it is executed every night by default
  2. The log file can no longer find the unfinished job status (running/savepoint)
  3. The log file has not been updated for 30 days, and users can change the configuration
  4. add testcase check this feature 5.add docs

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

corgy-w commented 4 days ago

can be assigned to me :>