search

text-machine-lab / TEA

Extracting narrative timelines (i.e. order and timing of events) from text
http://text-machine.cs.uml.edu/projects/temporal
20 stars 3 forks source link

Trouble with unicode character encodings... #3

Open mfeblowitz opened 6 years ago

mfeblowitz commented 6 years ago

Encountering errors in running predict_network.py, e.g., 

File "predict_network.py", line 327, in <module>
    main()
  File "predict_network.py", line 121, in main
    tmp_note = TimeNote(tml, tml)
  File "/homes/hny2/mfeb/causal/event-based-reasoning/code/notes/TimeNote.py", line 46, in __init__
    tokenized_text, token_to_offset, sentence_features, dependency_paths, id_to_tok = pre_processing.pre_process(self.original_text)
  File "/homes/hny2/mfeb/causal/event-based-reasoning/code/notes/utilities/pre_processing/pre_processing.py", line 26, in pre_process
    naf_tagged_doc = pre_processor.pre_process(text) # output a string of xml doc
  File "/homes/hny2/mfeb/causal/event-based-reasoning/code/notes/utilities/pre_processing/news_reader.py", line 68, in pre_process
    srl_text = srl.parse_dependencies(constituency_parsed_text)
  File "/homes/hny2/mfeb/causal/event-based-reasoning/code/notes/utilities/pre_processing/news_reader.py", line 124, in parse_dependencies
    return SRLClient.parse_dependencies(naf_tokenized_pos_tagged_text)
  File "/homes/hny2/mfeb/causal/event-based-reasoning/code/notes/utilities/pre_processing/news_reader.py", line 159, in parse_dependencies
    _output, _ = srl.communicate(naf_tokenized_pos_tagged_text)
  File "/homes/hny2/mfeb/anaconda3/envs/py27/lib/python2.7/subprocess.py", line 479, in communicate
    return self._communicate(input)
  File "/homes/hny2/mfeb/anaconda3/envs/py27/lib/python2.7/subprocess.py", line 1102, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/homes/hny2/mfeb/anaconda3/envs/py27/lib/python2.7/subprocess.py", line 1166, in _communicate_with_poll
    input_offset += os.write(fd, chunk)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 1787: ordinal not in range(128)

This is just one of many such errors.

Web postings indicate that the use of str() is the culprit in these cases, recommending the use of, e.g., .encode('utf-8'), instead. It's not clear, though, where the root cause might be. The trace indicates that it's writing the results of the parsing subprocess.

Any recommendations for addressing this?

ylmeng commented 6 years ago

Yes usually such problems are caused by encoding issues with some special characters. On the one hand, you can do some preprocessing to convert the unicode characters to ascii characters in all training and test files. On the other hand, you may try using encode('utf8') with all the unicode strings. For the model per se, such special characters are unimportant, so just do whatever you feel convenient.