search

xiangking / ark-nlp

A private nlp coding package, which quickly implements the SOTA solutions.
Apache License 2.0
310 stars 65 forks source link

convert_to_ids函数占用大量CPU #62

Closed lcao1210 closed 2 years ago

lcao1210 commented 2 years ago

您好，我利用样例以及GlobelPointerBert做NER任务加载数据时发现代码跑到convert_to_ids时，服务器CPU占用十分严重，通过htop查到服务器64核CPU占据了将约5000%，请问是什么原因呢

xiangking commented 2 years ago

您好，不好意思回迟了，convert_to_ids并没有做什么特别的操作，CPU占用高应该是torch默认占用的问题，可以通过torch.set_num_threads(1)等命令进行限制

lcao1210 commented 2 years ago

确实，添加限制之后CPU恢复正常，感谢！