search

chbrown / liwc-python

Linguistic Inquiry and Word Count (LIWC) analyzer
MIT License
193 stars 50 forks source link

Dealing with 'utf-8' encoding #15

Open murgado opened 3 years ago

murgado commented 3 years ago

Hello there! I'm dealing with a .dic file containing special characters such as "ñ" or "í", the problem lies whenever I do have to read such dictionary.

As an example: I do have the word "abadía" in my corpus and whenever I read it with liwc it appears as : "abadÃ\xada". How could I deal with this? Thank you very much in advance.

murgado commented 3 years ago

I've managed to solve this issue myself by accessing to the liwc/liwc/dic.py file and changing the read_dic(filepath) method to read my file using the utf-8 encoding.

vicru commented 3 years ago

I've managed to solve this issue myself by accessing to the liwc/liwc/dic.py file and changing the read_dic(filepath) method to read my file using the utf-8 encoding.

Thank you for sharing your own solution. I would appreciate if you provided the details about it. I just added encoding='utf-8' to liwc/liwc/dic.py, but apparently that is not right, because I keep getting "KeyError: '8'" with open(filepath, encoding='utf-8') as lines:

By the way, I am trying to use this library not with a Spanish but with a German (from 2001) version. That is why I think your suggested solution might make this library work with the German 2001 .dic

murgado commented 3 years ago

I've managed to solve this issue myself by accessing to the liwc/liwc/dic.py file and changing the read_dic(filepath) method to read my file using the utf-8 encoding.

Thank you for sharing your own solution. I would appreciate if you provided the details about it. I just added encoding='utf-8' to liwc/liwc/dic.py, but apparently that is not right, because I keep getting "KeyError: '8'" with open(filepath, encoding='utf-8') as lines:

By the way, I am trying to use this library not with a Spanish but with a German (from 2001) version. That is why I think your suggested solution might make this library work with the German 2001 .dic

That's exactly what we did, I noticed that by applying that tiny change our code worked but I'm not sure If maybe it should also be changed somewhere else. Have you debuggered your code?

vicru commented 3 years ago

I've managed to solve this issue myself by accessing to the liwc/liwc/dic.py file and changing the read_dic(filepath) method to read my file using the utf-8 encoding.

Thank you for sharing your own solution. I would appreciate if you provided the details about it. I just added encoding='utf-8' to liwc/liwc/dic.py, but apparently that is not right, because I keep getting "KeyError: '8'" with open(filepath, encoding='utf-8') as lines: By the way, I am trying to use this library not with a Spanish but with a German (from 2001) version. That is why I think your suggested solution might make this library work with the German 2001 .dic

That's exactly what we did, I noticed that by applying that tiny change our code worked but I'm not sure If maybe it should also be changed somewhere else. Have you debuggered your code?

Thank you very much for your quick reply. I actually haven't debugged the code. I don't find another solution for that, I will do that and post a reply.

By the way, I faced exactly the same situation with another LIWC python library and open an issue there