Expose full JobStatusResponse information for checking job status

[YuvalItzchakov]

Hi Yonatan,

When executing a request for driver status:

val submittionStatus: DriverState = client
  .checkJobStatus()
  .withSubmissionId(trackJobRequest.jobId)

We only get back the DriverState object which shows us the driver of the status. If you expose the full JobStatusResponse, one could take the location of the driver (to know which worker it runs on) and query Sparks driver API to get status regarding the actual running applications.

I'd be glad to supply a pull request which extends this, perhaps something along the lines of: withSubmissionIdFullStatus to the API so it won't be a breaking change.

Or perhaps, even a better approach would be to return this information when submitting the application, since Spark returns a full response like this:

{
  "action": "SubmissionStatusResponse",
  "driverState": "SUBMITTED",
  "serverSparkVersion": "2.0.2",
  "submissionId": "driver-20170128095903-0002",
  "success": true,
  "workerHostPort": "XXX.XXX.XXX.XXX:7078",
  "workerId": "worker-20170128095655-XXX.XXX.XXX.XXX-7078"
}

What do you think?

Regards, Yuval.

[ywilkof]

Hi Yuval, That sounds like a great idea! Looks like these details were exposed already two years ago, so I assume it's not even a Spark 2.0 addition (this client was tested as working from version 1.5 and up). Looking forward to your PR!

Yonatan

[ywilkof]

@YuvalItzchakov A new version incorporating your changes is now on maven central

[YuvalItzchakov]

Yay! Thanks @ywilkof :)