Currently, while calculating CoreComputeTime ( i.e. the "time-of-app" X "total-num-of-cores") , we include cores from all executors irrespective of how long an executor lived in the spark-app time-frame. This could give a blown-up number for CoreComputeTime available because executors could go down, and new executors could come up all the time.
On the other hand, finding the exact CoreComputeTime (i.e. summation across executors for ("time-of-executor" X "num-core-per-executor") ) is also not a good number, since it does not tell whether the machines had gone down along with executors or not (which is often 'not'). So the compute wasted on host-without-executors would not be accounted, if we only look at the executor up-time.
So a better number would be the "maximum-concurrent-executors-that-ran" X "time-of-app", which would give us a better estimate of the compute available.
beriaanirudh
commented
6 years ago
Currently, while calculating CoreComputeTime ( i.e. the "time-of-app" X "total-num-of-cores") , we include cores from all executors irrespective of how long an executor lived in the spark-app time-frame. This could give a blown-up number for CoreComputeTime available because executors could go down, and new executors could come up all the time.
On the other hand, finding the exact CoreComputeTime (i.e. summation across executors for ("time-of-executor" X "num-core-per-executor") ) is also not a good number, since it does not tell whether the machines had gone down along with executors or not (which is often 'not'). So the compute wasted on host-without-executors would not be accounted, if we only look at the executor up-time.
So a better number would be the "maximum-concurrent-executors-that-ran" X "time-of-app", which would give us a better estimate of the compute available.