For debugging/transparency, we should write an index file to S3 for a given job's inputs and outputs.
For ease of implementation, this doesn't need to be a singular file, we could have something like inputs.json and outputs.json which contain arrays of file names.
For debugging/transparency, we should write an index file to S3 for a given job's inputs and outputs.
For ease of implementation, this doesn't need to be a singular file, we could have something like
inputs.jsonandoutputs.jsonwhich contain arrays of file names.