apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.79k stars 1.75k forks source link

[Bug] [Seatunnel-Core] Too much flink-table-planner.jar in the tmp dir #7163

Open litiliu opened 2 months ago

litiliu commented 2 months ago

Search before asking

What happened

We are using a standalone cluster that doesn't exit to run the Seatunnel jobs submitted to it. After that cluster has been running for days, we found : There are lots of jars named flink-table-planner*.jar under the tmp folder, which has used up most of the available storage Screenshot 2024-07-08 at 17 52 58

This jar was created when the method org.apache.flink.table.planner.loader.PlannerModule#PlannerModule runs.

Screenshot 2024-07-11 at 10 56 42

SeaTunnel Version

2.3.5

SeaTunnel Config

{
  "env": {
    "job.mode": "BATCH",
    "parallelism": 1
  },
  "source": [
    {
      "plugin_name": "FakeSource",
      "result_table_name": "fake",
      "row.num": 100,
      "schema": {
        "fields": {
          "name": "string",
          "gender": "boolean",
          "card": "int"
        }
      }
    }
  ],
  "sink": [
    {
      "plugin_name": "Console",
      "source_table_name": "fake"
    }
  ]
}

Running Command

org.apache.seatunnel.example.flink.v2.SeaTunnelApiExample

Error Exception

There are lots of jars named `flink-table-planner*.jar` under the tmp folder, which has used up most of the available storage
![Screenshot 2024-07-08 at 17 52 58](https://github.com/apache/seatunnel/assets/38579068/238782b5-8033-49f5-9f54-6a1a3a31262f)

Zeta or Flink or Spark Version

flink 1.15.4

Java or Scala Version

java8

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

litiliu commented 2 months ago

Also in the flink 1.17 or higher version, that temp file will be deleted when the JVM exits, but in the case the JVM doesn't exit, the tmp dir will run out of storage. It's critical.

github-actions[bot] commented 4 weeks ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.