Spike - catch file encoding error?

[ADPennington]

Description: an aggregate file was rejected in TDP, and the error messages returned indicated the following:

After some testing, it turns out that the error messages are incorrect. the file actually failed because of encoding (after changing it to UTF-8, the file processed successfully.)

Open Questions: Please include any questions, possible solutions or decisions that should be explored during work

• Is it possible to catch this at the file upload stage and return a meaningful error message? https://github.com/raft-tech/TANF-app/blob/9b62b1cd5da401c5dce0135805b3ef8c2ce51a27/tdrs-frontend/src/components/FileUpload/FileUpload.jsx#L76
• if not, can this be caught as a pre-parsing error?

Deliverable(s): Create a list of recommendations or proofs of concept to be achieved to complete this issue

• [ ] Research blocking the file on-upload with nginx
• [ ] Research changing the file-encoding on-the-fly (if no data loss will occur)
• [ ] Craft some tests/integration tests using different file encodings

Supporting Documentation: Please include any relevant log snippets/files/screen shots

• Doc 1
• Doc 2

[ADPennington]

this issue came up again for tribal file submitted this week. an audit of rejected files needs to be conducted. suggested criteria for the audit:

• files submitted for FY24Q1 (last quarter)
• file status == Rejected
• error message includes Header length is 24...

cc: @lfrohlich @ysong001 @ttran-hub

[lhuxraft]

May want to explore accepting UTF-8 with BOM, potentially bypass the issue