[BUG] when using mulprocess

[phamkhactu]

Thanks for your great work,

when I using norm with multi process with pool: I get error:

[E] ucnv_toUChars: ICU Error "U_STRING_NOT_TERMINATED_WARNING"
[E] ucnv_toUChars: ICU Error "U_STRING_NOT_TERMINATED_WARNING"
568
249
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/tupk/anaconda3/envs/nlp/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/tupk/anaconda3/envs/nlp/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "data/mul_process_split_data.py", line 75, in split_data
    text_norm = TTSnorm(text)
  File "/home/tupk/anaconda3/envs/nlp/lib/python3.8/site-packages/vinorm/__init__.py", line 29, in TTSnorm
    text=fr.read()
  File "/home/tupk/anaconda3/envs/nlp/lib/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 69: invalid start byte
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "data/mul_process_split_data.py", line 92, in <module>
    print(p.map(split_data, files))
  File "/home/tupk/anaconda3/envs/nlp/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/tupk/anaconda3/envs/nlp/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 69: invalid start byte

I hop that you can check again it to make your package become supper start.

Best regards, Tu

[thucth-qt]

@phamkhactu did you happen to fix this issue? I have the same issue so I would appreciate that you could share a solution for this

[v-nhandt21]

Thank you for visiting, there are two solutions for this:

• I will update I/O for execute file -> Maybe when I have freetime :))

Hotfix:

• Could you try to write multiple sentences: https://github.com/v-nhandt21/Vinorm/blob/4a191ff79e8920c9293da1d8e1fdb985608b6576/vinorm/__init__.py#L10

  def TTSnorm(texts, punc = False, unknown = True, lower = True, rule = False ):
  A=imp.find_module('vinorm')[1]
  
  #print(A)
  I=A+"/input.txt"
  with open(I,"w+") as fw:
      for text in texts:
           fw.write(text + "\n")

• Handle output depend on your problem

  O=A+"/output.txt"
  with open(O,"r") as fr:
    texts_norm=fr.read().splitlines()
  return texts_norm

• Each in one line of input.txt file, and then manually run init.py

Feel free to comment for more discussion :))