A strange process with tranning

Sense-X / UniFormer

[ICLR2022] official implementation of UniFormer

Apache License 2.0

829 stars 111 forks source link

Closed LEM0NTE closed 2 years ago

LEM0NTE commented 2 years ago

您好，我在训练代码的时候，发现了奇怪的进程我不知道为什么device：0只占用了一半显存，而device：1有两个进程占满了显存。

同时，我在另外一台服务器上训练时进程显示似乎是正常的。

此外之前训练100个epoch大概需要七天左右。而这次训练预计时间却是13天，这可能是哪里出现了问题呢？希望得到您的答案~

Andy1621 commented 2 years ago

应该是你和别人共享了机器？CPU被强占了，会变慢的。见https://github.com/Sense-X/UniFormer/issues/64#issuecomment-1174710890

Andy1621 commented 2 years ago

又或者这个进程38030是个僵尸进程，和你7668/7670的进程不连续，你需要手动kill一下。

Andy1621 commented 2 years ago

As there is no more activity, I am closing the issue, don't hesitate to reopen it if necessary.