elvinpoon
commented
7 years ago For Q1: it's what this descirbed. windowed means that you put surrounding word together as input vector. For example, if you have sentence'开心啊' -> [1,2,3](refers to word index in vocabulary table), you will be transforming this sentence to [[B,1,2],[1,2,3],[2,3,E]] as input(B,E are padding index). Finally this will turn into [[embed(B)+embed(1)+embed(2)],[embed(1)+embed(2)+embed(3)],[embed(2)+embed(3)+embed(E)]].
For Q2: you can search online for 'CRF中文分词', refer to this article Here's the code you can use.
#!/usr/bin/env python
#-*-coding:utf-8-*-
#4-tags for character tagging: B(Begin),E(End),M(Middle),S(Single)
import codecs
import sys
def character_tagging(input_file, output_file):
input_data = codecs.open(input_file, 'r', 'utf-8')
output_data = codecs.open(output_file, 'w', 'utf-8')
for line in input_data.readlines():
word_list = line.strip().split()
for word in word_list:
if len(word) == 1:
output_data.write(word + "\tS\n")
else:
output_data.write(word[0] + "\tB\n")
for w in word[1:len(word)-1]:
output_data.write(w + "\tM\n")
output_data.write(word[len(word)-1] + "\tE\n")
output_data.write("\n")
input_data.close()
output_data.close()
if __name__ == '__main__':
if len(sys.argv) != 3:
print ("Usage: python " + argv[0] + " input output")
sys.exit(-1)
input_file = sys.argv[1]
output_file = sys.argv[2]
character_tagging(input_file, output_file)
lengyuewuyazui
Q1: What is windowed LSTM model?Is it the model that this paper presented?
Q2: How to transform the raw data like this 迈向 充满 希望 的 新 世纪 —— 一九九八年 新年 讲话 （ 附 图片 １ 张 ） to your training data-sets' form?