Split a whole data set into train dev and test

[mingfengwuye]

If I have a whole data set, is there a tool could split it into three parts, train dev and test? Thanks

[ghost]

You can do it manually, Always! Anyway, you can use packages like sklearn to split your data into train, test, evaluation (or dev).

[kmario23]

@mingfengwuye I heavily use commandline and bash to do this. I know it's a cumbersome process. Maybe you can write a shell script to do this more quickly. Anyway, for using new techniques like Subword-NMT, you've to preprocess data accordingly which I think can be done with simple custom shell script.

[oahziur]

@mingfengwuye After you split you data set with either sklearn or bath utils, you can reference this script to see how tokenization and subword is processed with existing tools.

[mingfengwuye]

Thank you all!

[ghost]

You should close the thread!