New tidyjobs handler that accepts a POST request. When called it will get a list of all running spark jobs and then write the logs to s3 for each completed or failed job. Once the logs are written the job will be deleted. Any job that is not in completed state remains untouched and if a failure occurs writing the logs to S3 then the job is not deleted.
The handler returns a list of spark applications with their statuses grouped by tidied, untouched or failed to process.
New
tidyjobshandler that accepts aPOSTrequest. When called it will get a list of all running spark jobs and then write the logs to s3 for each completed or failed job. Once the logs are written the job will be deleted. Any job that is not in completed state remains untouched and if a failure occurs writing the logs to S3 then the job is not deleted. The handler returns a list of spark applications with their statuses grouped bytidied,untouchedorfailed to process.Stories covered
96
To test