How to convert .txt file to .h5 file

guxd / deep-code-search

DeepCS: Deep Code Search

MIT License

278 stars 85 forks source link

Open XiaoXiaoYi123 opened 2 years ago

XiaoXiaoYi123 commented 2 years ago

ese /indices and /phrases?

li-car-fei commented 2 years ago

请问可以提供将txt文本序列转为对应的.h5文件的实现代码吗？

guxd commented 2 years ago

@li-car-fei https://github.com/guxd/DialogBERT/blob/master/prepare_data.py 参考这个，用pytable 把dialog (a list of sequences)转成earray.

li-car-fei commented 2 years ago

@li-car-fei https://github.com/guxd/DialogBERT/blob/master/prepare_data.py 参考这个，用pytable 把dialog (a list of sequences)转成earray.

在deepcs中，你是使用了哪里的tokenizer呢？

guxd commented 2 years ago

deepcs没有用第三方tokenizer,是自己实现的简易的tokenizer. 直接把字符转换成字典里的序号