User can call a writelogs handler with body {"job name": "example-job"}. The web app will get the logs for that job and write them to the kubernetes bucket in S3 at outputs/example-job/log.txt
Subsequent tickets will:
generalise this process to all jobs that have completed
improve the configuration
call the method on a cron job
Acceptance criteria
[x] A POST request to the new writelogs handler with body {"job_name": "{name of job}"}where the job exists produces a file in thekubernetesbucket in S3 atoutputs/example-job/log.txt`
[x] The contents of the produced log file matches those obtained by running kubectl logs {spark job name}-driver from a command window (with kubectl access to the kubernetes cluster)
[x] If the job does not exist a 404 is returned
[x] If "job_name" argument is missing from the request a 400 is returned with a message stating it is a required property - Fail. Error message include unrelevant details"data": "'job_name' is a required property\n\nFailed validating 'required' in schema:\n {'properties': {'job_name': {'type': 'string'}},\n 'required': ['job_name'],\n 'type': 'object'}\n\nOn instance:\n {}"
[x] The job is not deleted from the cluster after calling writelogs on it
[x] Calling the handler a second time for the same job overwrites the log file previously produced
Sub-task to #16
User can call a
writelogs
handler with body{"job name": "example-job"}
. The web app will get the logs for that job and write them to thekubernetes
bucket in S3 atoutputs/example-job/log.txt
Subsequent tickets will:
cron
jobAcceptance criteria
POST
request to the newwritelogs
handler with body {"job_name": "{name of job}"}where the job exists produces a file in the
kubernetesbucket in S3 at
outputs/example-job/log.txt`kubectl logs {spark job name}-driver
from a command window (with kubectl access to the kubernetes cluster)404
is returned400
is returned with a message stating it is a required property - Fail. Error message include unrelevant details"data": "'job_name' is a required property\n\nFailed validating 'required' in schema:\n {'properties': {'job_name': {'type': 'string'}},\n 'required': ['job_name'],\n 'type': 'object'}\n\nOn instance:\n {}"
writelogs
on it