Support for arbitrary file uploads

[adamkewley]

Asked for via email by a software developer who wants to use jobson to make a web interface for a bioinformatics tool.

Things needed (+ engineering considerations):

• Needs to support uploading arbitrary files (e.g. .fasta files)
  • Currently, hacked around by using a string[] input which, because of the various Gaia optimizations, does allow quite large inputs (e.g. I was uploading 100k+-elements via the UI at one point with no problems).
  • Ideally, should just allow the user to upload any file, so that non-text filetypes (e.g. gzippped fastq.gz files) can work also,
  • Allowing actual file types also will enable larger (multi-megabyte, at least) inputs because the UI can skip javascript trying to parse the input as a string array
• Needs to support uploading an arbitrary number of files (e.g. user can select a bunch of files from their computer via the UI)
  • Currently, no support for this in Jobson at all
  • Might initially require custom scripting functions in the templating language (e.g. ${toDirContainingFiles(filesInput)}) to handle the fact that the number of arguments passed to the application is currently static
  • e.g. Jobson "knows" that exactly 5 arguments are passed to the app because there are 5 entries in application: arguments: []. Ideally, this feature would also benefit from Jobson supporting runtime-determined number of arguments (because one use-case is to pass app file1 file2 ... filen to the underlying app)
  • Engineering concession: doing this properly (runtime number of arguments) might take a lot of time, because the test suites etc. assume arguments are static. One middle-ground solution to get something out the door, but not break forwards compatibility, would be to provide a function that dumps all the files into a single directory and then passes the directory as a single argument to the application, making the application responsible for searching the passed-in directory for files
• Files might be large (need to confirm this with the researcher). Jobson's backend implementation assumes a single JSON payload containing all data, which might explode the server's RAM if the user tries to upload multi-gigabyte base64-encoded JSON payloads etc.
  • If the files are large, then Jobson's API will need to be designed to accept multi-part requests, because the current design expects an entire JSON payload to be sent down the wire, parsed in Java, and then handled. A design that allows for large inputs would need to stream each part into storage as it arrives (to prevent memory from being blown up).
  • This is probably going to take a large amount of time to do properly (because I can only work on Jobson a few hours a week). Might need to recommend other approaches (e.g. putting the files on a network drive and then having the user send network-drive-paths via the Jobson UI: ugly, but would work)

Rough Tasklist

• [x] add support for file type in the backend, so that users can specify type: file in a job spec
  • Resulting Jobson-generated API should accept a { filename: Optional<string>, data: string }, where data is a b64-encoded data string
• [x] add support for file type in frontend, so that UI users can upload a file via the UI (which uses the file-generated API added to the backend)
  • Should use native browser File input support, which supports users selecting files on their local computer
  • Bonus: also supports drag + drop
• [x] add support for file[] type in backend, so that users can specify type: file[] in job spec
  • Resulting Jobson-generated API should accept a [{ filename: Optional<string>, data: string }] type, similar to above
• [x] add support for file[] type in frontend
  • Should use native browser File input support, as above
  • Should also try to use multi-select file input methods (e.g. drag + drop directories, multi-file-select-etc.)
• [ ] (Optional) support extension filtering
  • Allow dev to specify supportedExtensions in the job spec
  • Propagates to UI, only allowing users to upload files with specified suffix
• [ ] (Optional) support max file size limit
  • Allow dev to specify maxSize in the job spec
  • Prevents backend from accepting insanely-sized inputs

[adamkewley]

Alright, so the first step of this feature is now in this PR. Video of it in action here:

https://youtu.be/3FsZkDN-OJs

Needs documenting, hardening, etc., but it seems to mostly be working. I can ship this intermediate product as an undocumented (and, therefore, not "officially" supported - yet) feature, because shipping new versions of Jobson is quite easy (usually automated by travis).

[adamkewley]

Jobson 1.0.9 now includes the file support as an undocumented-but-working feature:

https://github.com/adamkewley/jobson/releases

[adamkewley]

This PR now houses implementing file[] support, which I've mostly done in the backend now, but it needs a relevant renderer in the UI (if you try it now, the UI will complain that it doesn't know what a file[] is).

[adamkewley]

The latest commit now has both backend and UI support for the file[] type. This implementation is a bit rough-and-ready (time constraints) but does work, mostly. Key points:

• Allows specifying file[] type as an expectedInput
• Renders a standard HTML5 <input type="file" multiple /> input as a consequence
• User can then select multiple files
• UI uses base64 encoding to embed the file content into a standard JSON message with this format:

{
    "spec": "specId",
    "inputs": {
        "inputId": [
            { "filename": "somefile.txt", "data": "<base64 string>" },
            { "filename": "someotherfile", "data": "<base 64 string>" }
        ]
    }
}

Backend accepts the JSON request and converts it (internally) back into a byte array + filename. Developers can then use toDir to put all of the files in into a single dir (duplicates are renamed accordingly):

# spec.yml

name: somespec
expectedInputs:
- id: filearray
  type: file[]
execution:
  application: echo
  arguments:
  - "${toDir(inputs.filearray)}"
  - "${inputs.filearray}"  # coerces into 'toDir(inputs.filearray)'

Which will give the runtime application a directory that it can then scan (e.g. with whatever directory listing function their language provides) to grab all inputs.

Things Missing

• Copying a job does not work, because the UI isn't sophisticated enough to know how to populate an <input type="file" /> from javascript
• For the same reasons, setting a default: arg for both file and file[] inputs will not populate the UI with the default on first loading
• No documentation for this feature
• No example usage of this feature
• No demo usage of this feature (e.g. an example generated with --demo)
• Because javascript ultimately has to slurp the entire file into memory, base64 it, and dump it into a single JSON payload (all in memory), large file (e.g. > 1GB, ~100 MB seems fine for me) inputs will crash the UI
• Because the entire data payload is in a single JSON object, the backend will need to read the entire thing into memory also, so the backend may crash if given extremely large inputs (because it'll try to parse the huge JSON payload into memory, run out of memory, then die)
• No support for file size limits, so this feature really shouldn't be used with untrusted actors
• No clear logging when a larger payload comes in, so that sysadmins can see which IP (guests) or user (logins) are causing problems
• No support for file extension filtering, so the UI will just accept any file
• No support for directory expansion (e.g. drag and drop a dir and all files are automatically added). Does support multi-selecting the file natively

[adamkewley]

Even with those problems/limitations in mind, I still think it would be a good idea to ship this as an undocumented feature. It's probably better to try this out and get the ball rolling than me spending weeks dealing with all this extra stuff. For developers using Jobson in smaller, closed deployments (e.g. research groups). They'll appreciate a somewhat-working feature today more than a perfect feature in acouple of months' time.