UnicodeDecodeError in UG

[axelboc]

I'm trying to run the source_to_anki.yml recipe in UG, but I'm running into this error:

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 5270: character maps to <undefined>

Here a few lines from the stack trace:

  File "c:\users\[...]\lib\site-packages\brain_brew\representation\generic\csv_file.py", line 34, in read_file
    self.column_headers = list_of_str_to_lowercase(csv_reader.fieldnames)
  File "C:\Program Files\Python37\lib\csv.py", line 98, in fieldnames
    self._fieldnames = next(self.reader)
  File "C:\Program Files\Python37\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]

Does it ring a bell?

[axelboc]

I'm on Windows (obviously the culprit), I've got both Python 3.8 and 3.7 installed. I created a Python 3.7 environment with pipenv as you explained in the CONTRIBUTING guide. I then run pipenv shell and brain_brew recipes\source_to_anki.yaml.

[aplaice]

Not testing on Windows is a pretty gigantic omission on my part. It somehow never crossed my mind...

Off the top of my head, I would guess that the issue is that Python on Windows uses the system locale encoding rather than UTF-8. It probably can be easily solved by explicitly specifying the encoding.

---

Presumably, adding , encoding='utf-8' in csv_file.py (l. 32):

    def read_file(self):
        self._data = []

        with open(self.file_location, mode='r', newline='', encoding='utf-8') as csv_file:
            csv_reader = csv.DictReader(csv_file)

            self.column_headers = list_of_str_to_lowercase(csv_reader.fieldnames)

if you're feeling adventurous and willing to meddle with the internal contents of a virtualenv, should solve this issue. (The encoding should probably also be specified in some other places, to prevent other bugs.)

[ohare93]

Does not ring a bell! I have run it on Windows, so this is surprising 🤔 I can have a look tomorrow morning (if @aplaice hasn't solved it already 😂👌)

[ohare93]

Apologies for the delay, will have a look at this on Saturday :+1: Should be a simple fix

[axelboc]

No rush, I'm swamped as well at the moment. Thanks, @aplaice for looking into it! ❤️

[ohare93]

Fixed. Thanks @aplaice :clap: