Open littleDrew opened 4 years ago
(1) You can use PythonTokenizer to achieve this function. see details in https://gitee.com/mindspore/mindspore/blob/master/mindspore/dataset/text/transforms.py and here is the test case to use PythonTokenizer: https://gitee.com/mindspore/mindspore/blob/master/tests/ut/python/dataset/test_python_tokenizer.py (2) You can also use WhitespaceTokenizer to split sentence by whitespace:
def test_whitespace_tokenizer():
"""
Test WhitespaceTokenizer
"""
whitespace_strs = [["Welcome", "to", "Beijing!"],
["北京欢迎您!"],
["我喜欢English!"],
[""]]
dataset = ds.TextFileDataset(DATA_FILE, shuffle=False)
tokenizer = nlp.WhitespaceTokenizer()
dataset = dataset.map(operations=tokenizer)
tokens = []
for i in dataset.create_dict_iterator():
text = nlp.to_str(i['text']).tolist()
tokens.append(text)
logger.info("The out tokens is : {}".format(tokens))
assert whitespace_strs == tokens
see https://gitee.com/mindspore/mindspore/blob/master/tests/ut/python/dataset/test_tokenizer.py for details.
Background
Introduction
map
function of datasetCurrenly in dataset, here support several kinds tokenizer like jiebaTokenizer and unicode tokenizer etal, and we can refer some use case of these to know how to use that.
unicodeTokenizer: https://github.com/mindspore-ai/mindspore/blob/master/tests/ut/python/dataset/test_tokenizer.py
jiebatokenizer: https://github.com/mindspore-ai/mindspore/blob/master/tests/ut/python/dataset/test_nlp_jieop.py
so how to develop a
simple_space_split
tokenizer in py_transform 1) its basic way, may like in pytorch :Do you have some implemented way and code ?