Some Spark runs are missing from Spot Elasticsearch indexes. It can be seen as gaps on the timeline of completed jobs.
This is likely happening when History server is catching up with previously completed runs (e.g. after a restart). In such scenario, new runs may appear in history in the order which is different from their completion order. However, Spot parses the jobs in their completion order: it receives a list of completed jobs within a time interval, the lower limit of the interval is updated in each iteration to match the max completion time of the previously processed jobs. As a result, if a job appears in Spark history later than some jobs which were completed after if will be skipped by Spot.
An investigation is required to confirm the issue and find possible solutions.
Some Spark runs are missing from Spot Elasticsearch indexes. It can be seen as gaps on the timeline of completed jobs.
This is likely happening when History server is catching up with previously completed runs (e.g. after a restart). In such scenario, new runs may appear in history in the order which is different from their completion order. However, Spot parses the jobs in their completion order: it receives a list of completed jobs within a time interval, the lower limit of the interval is updated in each iteration to match the max completion time of the previously processed jobs. As a result, if a job appears in Spark history later than some jobs which were completed after if will be skipped by Spot.
An investigation is required to confirm the issue and find possible solutions.