Make job folders robust to 3rd-party manipulation

[adamkewley]

This is for the sake of cronjobs / 3rd-party scripts / monkey patching a Jobson server.

The filesystem job persistence layer should allow:

• Arbitrary subdirs (e.g. .git): If the subdir does not contain a request.json, ignore it
• Arbitrary files (e.g. README): Ignored
• Deleting request.json: Should result in the job not being shown
• Deleting spec.json: Should result in the spec API 404ing and HATEOAS links being removed
• Deleting inputs.json: Should result in the inputs API 404ing and HATEOAS links being removed
• Deleting stderr: Should result in the stderr API 404ing and HATEOAS links being removed
• Deleting stdout: Should result in the stdout API 404ing and HATEOAS links being removed
• Deleting outputs/: Should result in no outputs appearing in the API
• Deleting an output (e.g. outputs/{output_id}): Should result in that particular output dissapearing from the API

[adamkewley]

• Implemented tests for each point
• Implemented fixes for all cases
• Not necessary for outputs/ (DAO already checks it)

TODO: Might be necessary for output metadata containing the entry but the actual data was manually deleted