API Discovery: How do we handle large, non-text outputs?

[srtalbot]

Goal

Identity how we would handle large, non-text outputs like PDF, XLSX, CSV, Images, or Doc files.

Acceptance criteria:

• [x] Outline the flow to scan for malicious files
• [ ] Identify how clients set file types to be accepted (e.g., The "upload a document" only allows PDFs)
• [ ] Files must be associated with a form response in a way that a machine can check automatically.
• [ ] ...

Related

• https://github.com/cds-snc/platform-forms-client/issues/934

[patheard]

I think there's a lot we can re-use from this design doc: Design Doc 008: File upload for API and UI + content scanning

Proposed API endpoints and behaviour would need to be adjusted, but the content scanning architecture and research into large file streaming should mostly hold with the caveat that we'd just need to retest the response content streaming using Express.js.

[srtalbot]

We're able to do basic steaming with Express, working on sending it piece by piece.

We need additional definition on what a large text file is. When is it too large to use Base-64.

@srtalbot to add research on file uploads

[srtalbot]

@craigzour - updated the design document with some requirements for file attachment.

https://docs.google.com/document/d/1gfnmZ3POdwd2nsIi6w6zgAgvyNnReK76vh2RHdYy35g/edit#heading=h.gyxo8mz7crtw

[srtalbot]

GC Notify provides a significant limitation to the file size.

We'll have to decide on how we approach file attachment.