[FEATURE] Client send sparkConf through registerApplicationInfo rpc to Coordinator

maobaolong commented 1 month ago

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before asking

[X] I have searched in the issues and found no similar issues.

Describe the feature

We can let client send sparkConf to coordinator to show more context information about this application.

Motivation

Insight and know what application are running and being served by RSS cluster.

Describe the solution

Create a util method like this.

  public static Map<String, String> sparkConfToMap(SparkConf sparkConf) {
    Map<String, String> map = new HashMap<>();

    for (Tuple2<String, String> tuple : sparkConf.getAll()) {
      String key = tuple._1;
      map.put(key, tuple._2);
    }

    return map;
  }

Extends the ApplicationInfoRequest

message ApplicationInfoRequest {
string appId = 1;
string user = 2;
// new added
map<string, string> properties = 3;
}

Other efforts related, finally display a column conf within application page.

Additional context

No response

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

jerqi commented 1 month ago

Application information should be displayed in the Spark history server.

maobaolong commented 1 month ago

@jerqi Sorry for that I do not explain more about the motivation.

Motivation I want to get the application relevant upstream trigger, in our scenacrio, it is livy, the upstream of livy is supersql and then is us.

If I have sparkConf, I can get these information by

usp.param=20240315170727772_20240726000000
spark.app.name=livy-session-23219-decb9ee5-a499-469a-b8ed-e23605e8efbf
spark.livy.supersql.session_id=decb9ee5-a499-469a-b8ed-e23605e8efbf
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES=http://xxxx/xxx

Base on these, we can get more context and know well about this job, we could dig the reason of failure or under performance.

Besides, we can get more useful information to help us for issue defeat.

spark.sql.warehouse.dir to check the cluster performance.
java version to check whether the java version is expected.
spark.internal.preload.jars to check whether the client jar is expected.
classpath to check whether the client jar is correct.
user.name to get the actually user name
spark.rss.* to check the effected config about uniffle.
spark.app.attempt.id to know the attempt time

apache / incubator-uniffle