search

apache / linkis

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
https://linkis.apache.org/
Apache License 2.0
3.32k stars 1.17k forks source link

feat: add support for sparkmeasure #5202

Open sjgllgh opened 1 week ago

sjgllgh commented 1 week ago

What is the purpose of the change

Add support for SparkMeasure to the Spark engine for better monitoring of Spark's performance.

Related issues/PRs

Related issues: #5200

How to use

  1. Here is an example of submitting parameters using RESTful API, with the specific parameters listed below: { "executionContent": { "code": "select * from test1.test1", "runType": "sql" }, "labels": { "engineType": "spark-3.2.1", "userCreator": "zhangyuyao-IDE" }, "params": { "configuration": { "runtime": { "linkis.sparkmeasure.aggregate.type": "stage" }, "startup": { "linkis.sparkmeasure.flight.recorder.type": "task" } } } }
  2. It is only valid for some DQLs, such as SELECT, INSERT, and CREATE AS SELECT.
  3. Indicators can be categorized into aggregate indicators and detailed indicators, and each type of indicator can further be divided into two sub-categories: stage and task.
  4. For aggregate indicators (using the parameter: linkis.sparkmeasure.aggregate.type), each eligible SQL query will have a separate file output.
  5. For detailed indicators (using the parameter: linkis.sparkmeasure.flight.recorder.type), an engine will only output one file, and the file will be generated when the engine is shut down. It is recommended to only execute one SQL statement per engine when using this feature.
  6. Due to the reuse mechanism of the Linkis engine, using linkis.sparkmeasure.flight.recorder.type does not necessarily result in the creation of a new engine. If an existing engine is reused, it may lead to no indicator file being output.
  7. To ensure that the file is output to the correct path, you should add the following parameter to the Spark engine: linkis.sparkmeasure.output.prefix.
  8. Currently, support is provided for writing files to both local and HDFS storage.
  9. For a detailed introduction to sparkmeasure, please refer to SparkMeasure

Note

  1. The files generated by SparkMeasure Flight Recorder can be relatively large.

Checklist