UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 360: character maps to <undefined>

[narasimha1805]

Getting 'UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 360: character maps to ' error while importing word_topic_vectors from nlpia.book.examples.ch04_catdog_las*

Below is the error:

UnicodeDecodeError Traceback (most recent call last)

in ----> 1 from nlpia.book.examples.ch04_catdog_lsa_3x6x16 import word_topic_vectors d:\python\lib\site-packages\nlpia\book\examples\ch04_catdog_lsa_3x6x16.py in 68 tfidfer = TfidfVectorizer(min_df=2, max_df=.6, stop_words=None, token_pattern=r'(?u)\b\w+\b') 69 ---> 70 corpus = get_data('cats_and_dogs')[:NUM_DOCS] 71 docs = normalize_corpus_words(corpus, stemmer=None) 72 tfidf_dense = pd.DataFrame(tfidfer.fit_transform(docs).todense()) d:\python\lib\site-packages\nlpia\loaders.py in get_data(name, nrows, limit) 1111 return filepaths[name] 1112 elif name in DATASET_NAME2FILENAME: -> 1113 return read_named_csv(name, nrows=nrows) 1114 elif name in DATA_NAMES: 1115 return read_named_csv(DATA_NAMES[name], nrows=nrows) d:\python\lib\site-packages\nlpia\loaders.py in read_named_csv(name, data_path, nrows, verbose) 1003 name = DATASET_NAME2FILENAME[name] 1004 if name.lower().endswith('.txt') or name.lower().endswith('.txt.gz'): -> 1005 return read_text(os.path.join(data_path, name), nrows=nrows) 1006 else: 1007 return read_csv(os.path.join(data_path, name), nrows=nrows) d:\python\lib\site-packages\nlpia\futil.py in read_text(forfn, nrows, verbose) 416 """ 417 tqdm_prog = tqdm if verbose else no_tqdm --> 418 nrows = wc(forfn, nrows=nrows) # not necessary when nrows==None 419 lines = np.empty(dtype=object, shape=nrows) 420 with ensure_open(forfn) as f: d:\python\lib\site-packages\nlpia\futil.py in wc(f, verbose, nrows) 48 tqdm_prog = tqdm if verbose else no_tqdm 49 with ensure_open(f, mode='r') as fin: ---> 50 for i, line in tqdm_prog(enumerate(fin)): 51 if nrows is not None and i >= nrows - 1: 52 break d:\python\lib\encodings\cp1252.py in decode(self, input, final) 21 class IncrementalDecoder(codecs.IncrementalDecoder): 22 def decode(self, input, final=False): ---> 23 return codecs.charmap_decode(input,self.errors,decoding_table)[0] 24 25 class StreamWriter(Codec,codecs.StreamWriter): UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1592: character maps to

[woo9904]

It is a UnicodeDecodeError. Maybe this example can make some help for understanding about the error.

file = open(filename, encoding="utf8")

For a solution, find futil.py file installed in your computer. (d:\python\lib\site-packages\nlpia\futil.py)

find function named ensure_open and edit some code.

fin = f
if isinstance(f, basestring):
    if len(f) <= MAX_LEN_FILEPATH:
        f = find_filepath(f) or f
        if f and (not hasattr(f, 'seek') or not hasattr(f, 'readlines')):
            if f.lower().endswith('.gz'):
                return gzip.open(f, mode=mode,encoding='UTF-8')
            return open(f, mode=mode,encoding='UTF-8')
        f = fin  # reset path in case it is the text that needs to be opened with StringIO
    else:
        f = io.StringIO(f)
elif f and getattr(f, 'closed', None):
    if hasattr(f, '_write_gzip_header'):
        return gzip.open(f.name, mode=mode,encoding='UTF-8')
    else:
        return open(f.name, mode=mode,encoding='UTF-8')
return f

I just added ",encoding='UTF-8'" when open() comes out.

[danielgran]

Doesn't work for me either, whats the problem?

[danielgran]

It is a UnicodeDecodeError. Maybe this example can make some help for understanding about the error.

file = open(filename, encoding="utf8")

For a solution, find futil.py file installed in your computer. (d:\python\lib\site-packages\nlpia\futil.py)

find function named ensure_open and edit some code.

fin = f
if isinstance(f, basestring):
    if len(f) <= MAX_LEN_FILEPATH:
        f = find_filepath(f) or f
        if f and (not hasattr(f, 'seek') or not hasattr(f, 'readlines')):
            if f.lower().endswith('.gz'):
                return gzip.open(f, mode=mode,encoding='UTF-8')
            return open(f, mode=mode,encoding='UTF-8')
        f = fin  # reset path in case it is the text that needs to be opened with StringIO
    else:
        f = io.StringIO(f)
elif f and getattr(f, 'closed', None):
    if hasattr(f, '_write_gzip_header'):
        return gzip.open(f.name, mode=mode,encoding='UTF-8')
    else:
        return open(f.name, mode=mode,encoding='UTF-8')
return f

I just added ",encoding='UTF-8'" when open() comes out.

Unfortunately that prints this error: File "gensim/_matutils.pyx", line 1, in init gensim._matutils ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

[danielgran]

It is a UnicodeDecodeError. Maybe this example can make some help for understanding about the error. file = open(filename, encoding="utf8") For a solution, find futil.py file installed in your computer. (d:\python\lib\site-packages\nlpia\futil.py) find function named ensure_open and edit some code.

fin = f
if isinstance(f, basestring):
    if len(f) <= MAX_LEN_FILEPATH:
        f = find_filepath(f) or f
        if f and (not hasattr(f, 'seek') or not hasattr(f, 'readlines')):
            if f.lower().endswith('.gz'):
                return gzip.open(f, mode=mode,encoding='UTF-8')
            return open(f, mode=mode,encoding='UTF-8')
        f = fin  # reset path in case it is the text that needs to be opened with StringIO
    else:
        f = io.StringIO(f)
elif f and getattr(f, 'closed', None):
    if hasattr(f, '_write_gzip_header'):
        return gzip.open(f.name, mode=mode,encoding='UTF-8')
    else:
        return open(f.name, mode=mode,encoding='UTF-8')
return f

I just added ",encoding='UTF-8'" when open() comes out.

Unfortunately that prints this error: File "gensim/_matutils.pyx", line 1, in init gensim._matutils ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

ah, nvm. this fixes it. thank you!

[hsang]

It is a UnicodeDecodeError. Maybe this example can make some help for understanding about the error.

file = open(filename, encoding="utf8")

For a solution, find futil.py file installed in your computer. (d:\python\lib\site-packages\nlpia\futil.py)

find function named ensure_open and edit some code.

fin = f
if isinstance(f, basestring):
    if len(f) <= MAX_LEN_FILEPATH:
        f = find_filepath(f) or f
        if f and (not hasattr(f, 'seek') or not hasattr(f, 'readlines')):
            if f.lower().endswith('.gz'):
                return gzip.open(f, mode=mode,encoding='UTF-8')
            return open(f, mode=mode,encoding='UTF-8')
        f = fin  # reset path in case it is the text that needs to be opened with StringIO
    else:
        f = io.StringIO(f)
elif f and getattr(f, 'closed', None):
    if hasattr(f, '_write_gzip_header'):
        return gzip.open(f.name, mode=mode,encoding='UTF-8')
    else:
        return open(f.name, mode=mode,encoding='UTF-8')
return f

I just added ",encoding='UTF-8'" when open() comes out.

Thanks, it works!