anaderi / skygrid

Prototype project for SHIP grid-cloud framework developed in the context of CERN openlab summer school
2 stars 0 forks source link

To determine is it possible to get cluster capabilities from Application Master #9

Closed xni closed 10 years ago

xni commented 10 years ago

Yes, It is. That's my application master's code:

public void run() throws YarnException, IOException {
    amRMClient = AMRMClientAsync.createAMRMClientAsync(1000, new RMCallbackHandler());
    amRMClient.init(conf);
    amRMClient.start();

    RegisterApplicationMasterResponse response;
    response = amRMClient.registerApplicationMaster(NetUtils.getHostname(), -1, "");
    LOG.info("ApplicationMaster is registered with response: {}", response.toString());

    // As a client!
    YarnConfiguration conf = new YarnConfiguration();
    YarnClient yarnClient = YarnClient.createYarnClient();
    yarnClient.init(conf);
    yarnClient.start();
    YarnClusterMetrics clusterMetrics = yarnClient.getYarnClusterMetrics();
    LOG.info("Cluster metrics is: " + clusterMetrics.toString());
    List<NodeReport> nodeReports = yarnClient.getNodeReports();
    for (NodeReport report :nodeReports) {
        LOG.info("Report: " + report.toString());
    }

    yarnClient.stop();
    nmClient.stop();
    amRMClient.unregisterApplicationMaster(FinalApplicationStatus.SUCCEEDED, "Application complete!", null);
    amRMClient.stop();
}

And the report looks like

14/07/27 17:38:32 INFO yarntest.ApplicationMaster: Cluster metrics is: num_node_managers: 1
14/07/27 17:38:32 INFO yarntest.ApplicationMaster: Report: nodeId { host: "master.local" port: 45091 } httpAddress: "master.local:8042" rackName: "/default-rack" used { memory: 6144 virtual_cores: 6 } capability { memory: 8192 virtual_cores: 8 } numContainers: 6 node_state: NS_RUNNING health_report: "" last_health_report_time: 1406482691685

So, total amount of used memory and cores is a good metrics to make a decision, if a new task should be taken.