NickleDave
commented
3 years ago Hi @marichard123 thank you for raising a clear detailed issue -- sorry you're having this problem.
We are very excited to work with a computer scientist that would actually think to look at the encoding used 😁 but I think your hunch is right, that a good place to start is hunting down the offending file.
Before doing that: are you able to provide a little bit more information about your annotation file(s)? Is there a single .csv file with all the annotations for every audio file, or is it one annotation file per audio file?
I'm wondering if a quick fix is to simply save the original file(s) in a different encoding. If you can tell me more about how you generated them, that might help us figure it out.
I can see that the crash occurred when crowsetta tried to open one of them.
File "c:\users\richard\anaconda3\envs\vak-env\lib\site-packages\crowsetta\csv.py", line 220, in csv2annot
set_header = set(reader.fieldnames)
File "c:\users\richard\anaconda3\envs\vak-env\lib\csv.py", line 98, in fieldnames
self._fieldnames = next(self.reader)
File "c:\users\richard\anaconda3\envs\vak-env\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]If you are okay with sharing annotation files either here or by email, I am also happy to run them through crowsetta to see if I can diagnose what's going on.
It might be the case that we need to do a little work on the crowsetta code to make csv loading more general -- I'd really appreciate if you can help me figure that out before we have a ton of users testing things and getting angry about encoding errors 😅
It's probably better to use something than our own hand-rolled csv read/write code anyway (as discussed in this issue)
As far as finding the file: A quick way to troubleshoot might be making a dataset with a single audio file and see if it still crashes.
It looks like the crash happened right at the start of checking annotations, so it might not be worth going to all this trouble.
But if I was going to try and track down the file, I would set a breakpoint with pdb as far down the stack I can get.
In this case, that looks like it's inside c:\users\richard\anaconda3\envs\vak-env\lib\site-packages\crowsetta\csv.py.
Specifically this line:
https://github.com/NickleDave/crowsetta/blob/3db98293b7babd1526dc0f28d7919181ec5b591a/src/crowsetta/csv.py#L207
with open(csv_filename, 'r', newline='') as csv_file:
reader = csv.DictReader(csv_file)
# DictReader automatically uses first row (AKA 'header') as fieldnames
# when no argument supplied for fieldnames parameter
# so we use that default to check validity of csv fieldnames
set_header = set(reader.fieldnames)
if set_header != set(CSV_FIELDNAMES):I would edit the file to look something like this:
with open(csv_filename, 'r', newline='') as csv_file:
reader = csv.DictReader(csv_file)
# DictReader automatically uses first row (AKA 'header') as fieldnames
# when no argument supplied for fieldnames parameter
# so we use that default to check validity of csv fieldnames
try:
set_header = set(reader.fieldnames)
except:
import pdb;pdb.set_trace()
if set_header != set(CSV_FIELDNAMES):and then when you get an error, just show the filename from the pdb prompt, e.g.,
(Pdb) p `the bane of our existence: ` + csv_filename
'the bane of our existence: ./data/some-annotation.csv'Of course be careful with editing the files in site packages since you can't set them back to the originals with git checkout or anything. You might also want to double check that you fixed the encoding back inside the cp1252.py file so that's not causing some unexpected errors--you definitely shouldn't have to touch that file!
If you're getting really weird errors it might be worth just re-creating the environment from scratch.
Please do let me know what you can about the file types and we can take it from there.
marichard123
When running the prep stage, at the point after which I believe the spectrograms are created, I get the following error message (I have attached the full error traceback at the end of the message):
File "c:\users\richard\anaconda3\envs\vak-env\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 164: character maps to
Initially I thought it was a problem of Python not being set to the correct encoding standard, so inside the cp1252.py file I tried manually setting the encoding procedure to ANSI and UTF-8, which the created text files were created in, with no success. I then noticed that in the second half of the file there was a decoding table, a tuple in which the encoding codes were all manually listed.
Among them were codes mapping to "undefined". Byte 0x90 indeed maps to 'undefined'. In other words, it seems to me that rather than a case of 0x90 not being defined in whatever encoding procedure Python is using due to encoding mismatch, that 0x90 is hard-coded to map to "undefined", and that the problem lies within whatever file the program is reading from. I'm not sure how to identify the file that's causing the problem/how to pinpoint what exactly is causing the 0x90 to appear in it. Have you run into a similar error during development/would you have any insight into the nature of the problem?
Full error traceback: