a write logs handler that takes a job name as input and writes the logs for that job to a standard location on S3
Extras:
removing /etc/boto.cfg in .travis.yml is necessary for the Travis build (otherwise the Travis VM tries to load unneeded packages that it cannot find)
separated the unit tests for SparkJobService by the method called
a little pylint tidying
Left to do in future PRs:
make the log output location more configurable
a method for writing logs of all completed jobs that don't yet have their logs written, then deleting the completed jobs
a cron-job caller of this method
Note that SecretsDir and IsS3Secure have been added to the congfiguration file and so deployment configs will need to be updated once this PR is merged in.
Main changes:
Extras:
/etc/boto.cfgin.travis.ymlis necessary for the Travis build (otherwise the Travis VM tries to load unneeded packages that it cannot find)SparkJobServiceby the method calledpylinttidyingLeft to do in future PRs:
Note that
SecretsDirandIsS3Securehave been added to the congfiguration file and so deployment configs will need to be updated once this PR is merged in.