gwu-libraries / batch-loader

Application for batch loading GW ScholarSpace
MIT License
1 stars 4 forks source link

batch-loader.py fails on csv with diacritical marks #18

Open reliztrent opened 5 years ago

reliztrent commented 5 years ago

Attempt to upload a csv with "Émigré" resulted in the following error. The error was resolved by replacing "Émigré" with "Emigre" in the csv. As a workaround, we were able to use the admin ui editor to re-insert the correct text.

$python batch_loader.py /path/to/file.csv
Traceback (most recent call last):
  File "batch_loader.py", line 152, in <module>
    field_names, rows = load_csv(args.csv)
  File "batch_loader.py", line 30, in load_csv
    return reader.fieldnames, list(reader)
  File "/usr/lib/python3.5/csv.py", line 96, in fieldnames
    self._fieldnames = next(self.reader)
  File "/opt/batch-loader/ENV/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4588: ordinal not in range(128)

Corrected item in prod: https://scholarspace.library.gwu.edu/concern/gw_works/z316q1971?locale=en