search

Jittor / jrender

Jrender is an efficient differentiable rendering library implemented in jittor.
315 stars 31 forks source link

performance of nerf #7

Open KayChou opened 2 years ago

KayChou commented 2 years ago

I train demo7-nerf.py in one GPU (3080ti), but only got 2.5 iterations per second. But this repo claims that jittor i s able to achieve 10+ its/s. I‘m wondering if I need to do special configuration? And what's the official test result?

Snipaste_2022-04-23_14-11-45

Also the newest commit seems to has some bug when entering block "if i%args.i_testset==0 and i > 0:”

Jittor commented 2 years ago

您好！感谢反馈！

Jrender NeRF每秒能迭几次跟数据集的config文件有关系，具体速度会受到batch size，划分chunk数等参数影响。

我们在单张NVIDIA-TITAN-RTX显卡（CUDA 11.6 Driver 510.47 jittor 1.3.3.5）上，用config/Easyship.txt 做测试

iter

速度为7.4 iter/s，如上图所示。 train test

训练显存为6.3G，测试显存为4.2G。

"if i%args.i_testset==0 and i > 0:” 确实是我们之前merge代码的时候出现的bug，感谢您的反馈，已经解决，请pull最新的仓库更新代码。您可以把本机测试的环境发给我们，我们来一起检查是什么问题。

KayChou commented 2 years ago

你好，我们做了测试，系统ubuntu18.04，单张3090，（CUDA 11.1 Driver 455.23 jittor 1.3.2.7）。

在前300个iterations时候速度可以达到 10+ its/s，如下图

Snipaste_2022-04-23_19-59-46

但是随着迭代数增加（大约1000 its 以后），速度逐渐下降到 2-3 its/s，此后基本上保持这个速度不变

Snipaste_2022-04-23_20-02-16
Yannnnnnnnnnnn commented 2 years ago

可能要更新一下jittor，都更新到1.3.3.8了。 我没有遇到这个效率下降的问题。 image

Jittor commented 2 years ago

@KayChou 同学你好，你的GPU Fan已经到了100%，显卡温度也已经烧到了90度，再高几度显卡就要报废了。应该是在300次迭代之后，显卡温度过高GPU主动降频了，建议检查一下散热。可以运行一下nvidia-smi -a会输出降频的理由。

此外建议把jittor升级到最新版，最新版在显存、速度上有优化。

Jittor commented 2 years ago

please check the memory usage，if memory usage is overflowed(nvidia-smi show gpu memory is 100% fully used), the performance will slow down. or you can set enviroment variable 'export use_cuda_managed_allocator=0'

---Original--- From: @.> Date: Sat, Apr 23, 2022 14:15 PM To: @.>; Cc: @.***>; Subject: [Jittor/jrender] performance of nerf (Issue #7)

I train demo7-nerf.py in one GPU (3080ti), but only got 2.5 iterations per second. But this repo claims that jittor i s able to achieve 10+ its/s. I‘m wondering if I need to do special configuration? And what's the official test result?

Also the newest commit seems to has some bug when entering block "if i%args.i_testset==0 and i > 0:”

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>