[dataset] how to inplement a simpler kinds of tokenizer: simple_space_split

[littleDrew]

Background

• here in mindspore, there is text part in dataset for doing data_augmentation in some nlp task
• in this process, we need some tokenizer to deal with origin text data, firstly tokenizer is needed

Introduction

• To better support all kinds of tokenizer, we might can implement it in py_transform
• For instruction, py_transform can be consider as a wraped python function, and easy to call in map function of dataset

• Currenly in dataset, here support several kinds tokenizer like jiebaTokenizer and unicode tokenizer etal, and we can refer some use case of these to know how to use that.

  • unicodeTokenizer: https://github.com/mindspore-ai/mindspore/blob/master/tests/ut/python/dataset/test_tokenizer.py

  • jiebatokenizer: https://github.com/mindspore-ai/mindspore/blob/master/tests/ut/python/dataset/test_nlp_jieop.py

  • so how to develop a simple_space_split tokenizer in py_transform 1) its basic way, may like in pytorch :

    • https://pytorch.org/text/data_functional.html#simple-space-split
    • https://pytorch.org/text/_modules/torchtext/data/functional.html#simple_space_split 2) as for how to write a py_transform:
      • we can refer some already done case, like OneHotOp: https://github.com/mindspore-ai/mindspore/blob/master/mindspore/dataset/transforms/py_transforms.py

Do you have some implemented way and code ?

[qianlong21st]

(1) You can use PythonTokenizer to achieve this function. see details in https://gitee.com/mindspore/mindspore/blob/master/mindspore/dataset/text/transforms.py and here is the test case to use PythonTokenizer: https://gitee.com/mindspore/mindspore/blob/master/tests/ut/python/dataset/test_python_tokenizer.py (2) You can also use WhitespaceTokenizer to split sentence by whitespace:

def test_whitespace_tokenizer():
    """
    Test WhitespaceTokenizer
    """
    whitespace_strs = [["Welcome", "to", "Beijing!"],
                       ["北京欢迎您！"],
                       ["我喜欢English!"],
                       [""]]
    dataset = ds.TextFileDataset(DATA_FILE, shuffle=False)
    tokenizer = nlp.WhitespaceTokenizer()
    dataset = dataset.map(operations=tokenizer)
    tokens = []
    for i in dataset.create_dict_iterator():
        text = nlp.to_str(i['text']).tolist()
        tokens.append(text)
    logger.info("The out tokens is : {}".format(tokens))
    assert whitespace_strs == tokens

see https://gitee.com/mindspore/mindspore/blob/master/tests/ut/python/dataset/test_tokenizer.py for details.