Automatically handle notifications upload files not encoded with UTF-8

[jimleroyer]

Description

As a user preparing a CSV upload, I would like to upload my files with minimal friction, especially when these come out of different programs with different character set encoding, So that I can send notifications as easy as possible with no blockers, in a transparent manner.

WHY are we building?

Users had bad experience uploading a file with a Windows coding page such as CP-1252. Notify rejects accents specified in that character set without much explanation.

WHAT are we building?

An automatic conversion to UTF-8 when detection of another coding page is used. Also better validation message to the user when such automatic conversion is not possible.

VALUE created by our solution

Better user experience and less support.

Acceptance Criteria** (Definition of done)

QA Steps

• [ ] Tested in a realistic production scenario

Additional info

The CSV upload error might trigger from this point in code:

• https://github.com/cds-snc/notification-admin/blob/df0aa4ac92bc54ecf0e7511f17274a42eecf127e/app/main/views/send.py#L172

A few stackoverflow entries on potential solutions:

• https://stackoverflow.com/questions/191359/how-to-convert-a-file-to-utf-8-in-python
• https://codereview.stackexchange.com/a/202985/121050

The chardet Python module is mentioned many times. There is also a confidence score we can get out of the file detection with this module which could be handy.

[yaelberger-commits]

Can we bring this forward for refinement/estimation discussion this Wednesday @jimleroyer to help make a discussion about path forward, tech solution or documentation? We can discuss the two cards together, this and #792