search

SYSU-STAR / H2-Mapping

H2-Mapping: Real-time Dense Mapping Using Hierarchical Hybrid Representation (2023 RAL Best Paper Award)
GNU General Public License v3.0
379 stars 15 forks source link

loss曲线抖动幅度太大 #20

Closed small-zeng closed 8 months ago

small-zeng commented 10 months ago

当我把loss曲线画出来后，发现所有loss抖动都很大，rgb loss后来还上升了，这个是正常的吗（但是重建效果还可以，是什么原因呢）？下面分别是两组数据：

Replica room0： image

自己采集的L515数据： image image

small-zeng commented 10 months ago

这个loss我看到rgb loss上升了，为什么图像质量反而提高了，难道还有其他因素吗？

JIANG-CX commented 10 months ago

看起来还是正常的，因为这是一个online的incremental的训练过程，所以画出来的loss是当前训练区域的loss情况，会受到如“当前训练视角新区域的占比”以及“当前区域的复杂程度、训练难度”的影响，比如replica可以看到会有上升的情况，主要是因为看到了较多的新场景，所以导致loss变大。此外我发现现在的结果比之前的mesh好很多，请问是做了哪些调整呢？

small-zeng commented 10 months ago

这个是因为采集的depth被我resize之后，depth出现了噪声，用原始分辨率就没有问题了

JIANG-CX commented 10 months ago

ok

获取 Outlook for iOShttps://aka.ms/o0ukef

发件人: Jing Zeng @.> 发送时间: Thursday, November 30, 2023 2:13:11 PM 收件人: SYSU-STAR/H2-Mapping @.> 抄送: JIANG Chenxing @.>; Comment @.> 主题: Re: [SYSU-STAR/H2-Mapping] loss曲线抖动幅度太大 (Issue #20)

这个是因为采集的depth被我resize之后，depth出现了噪声，用原始分辨率就没有问题了

― Reply to this email directly, view it on GitHubhttps://github.com/SYSU-STAR/H2-Mapping/issues/20#issuecomment-1833167525, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APMKKNR26LOXN73OCTKDHU3YHAPXPAVCNFSM6AAAAAA77NXUYKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZTGE3DONJSGU. You are receiving this because you commented.Message ID: @.***>

small-zeng commented 10 months ago

你好, 最近我在训练replica room0的时候，显存会逐渐增加，mapping 我重新clone 一份还是这样，这个问题出现在哪里呢： $ python -W ignore demo/run_mapping.py configs/replica/room_0.yaml *** initializing first_frame: 0**** mapping started! mapping frame: 44%|████████████████████████████████████████████████████████████████████▉ | 877/1999 [03:46<04:49, 3.88it/s] Traceback (most recent call last): File "demo/run_mapping.py", line 26, in slam.run() File "/mnt/dataset/zengjing/H2-Mapping/mapping/src/H2Mapping.py", line 26, in run self.mapper.run(self.firstframe, update_pose=self.update_pose) File "/mnt/dataset/zengjing/H2-Mapping/mapping/src/mapping.py", line 271, in run self.mapping_step(frame_id, tracked_frame, update_pose) File "/mnt/dataset/zengjing/H2-Mapping/mapping/src/mapping.py", line 183, in mapping_step self.do_mapping(tracked_frame=tracked_frame, update_pose=update_pose) File "/mnt/dataset/zengjing/H2-Mapping/mapping/src/mapping.py", line 310, in do_mapping bundle_adjust_frames( File "/mnt/dataset/zengjing/H2-Mapping/mapping/src/functions/render_helpers.py", line 472, in bundle_adjust_frames sampled_xyz = ray(rays_o_all[0].unsqueeze( File "/mnt/dataset/zengjing/H2-Mapping/mapping/src/functions/render_helpers.py", line 22, in ray return ray_start + ray_dir * depths torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 23.70 GiB total capacity; 19.74 GiB already allocated; 10.69 MiB free; 21.05 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF (sdfstudio) zju@zju-FitServer-G4460-V5:/mnt/dataset/zengjing/H2-Mapping/mapping$

JIANG-CX commented 10 months ago

应该是因为保存的keyframe逐渐增加了，但看起来你是24G显存？应该不会溢出来着

获取 Outlook for iOShttps://aka.ms/o0ukef

发件人: Jing Zeng @.> 发送时间: Thursday, November 30, 2023 2:23:54 PM 收件人: SYSU-STAR/H2-Mapping @.> 抄送: JIANG Chenxing @.>; Comment @.> 主题: Re: [SYSU-STAR/H2-Mapping] loss曲线抖动幅度太大 (Issue #20)

你好, 最近我在训练replica room0的时候，显存会逐渐增加，mapping 我重新clone 一份还是这样，这个问题出现在哪里呢： $ python -W ignore demo/run_mapping.py configs/replica/room_0.yaml * initializing first_frame: 0**** mapping started! mapping frame: 44%|€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€ | 877/1999 [03:46<04:49, 3.88it/s] Traceback (most recent call last): File "demo/run_mapping.py", line 26, in slam.run() File "/mnt/dataset/zengjing/H2-Mapping/mapping/src/H2Mapping.py", line 26, in run self.mapper.run(self.firstframe, update_pose=self.update_pose) File "/mnt/dataset/zengjing/H2-Mapping/mapping/src/mapping.py", line 271, in run self.mapping_step(frame_id, tracked_frame, update_pose) File "/mnt/dataset/zengjing/H2-Mapping/mapping/src/mapping.py", line 183, in mapping_step self.do_mapping(tracked_frame=tracked_frame, update_pose=update_pose) File "/mnt/dataset/zengjing/H2-Mapping/mapping/src/mapping.py", line 310, in do_mapping bundle_adjust_frames( File "/mnt/dataset/zengjing/H2-Mapping/mapping/src/functions/render_helpers.py", line 472, in bundle_adjust_frames sampled_xyz = ray(rays_o_all[0].unsqueeze( File "/mnt/dataset/zengjing/H2-Mapping/mapping/src/functions/render_helpers.py", line 22, in ray return ray_start + ray_dir * depths torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 23.70 GiB total capacity; 19.74 GiB already allocated; 10.69 MiB free; 21.05 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF (sdfstudio) **@.***:/mnt/dataset/zengjing/H2-Mapping/mapping$

― Reply to this email directly, view it on GitHubhttps://github.com/SYSU-STAR/H2-Mapping/issues/20#issuecomment-1833178010, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APMKKNXLCNEEL4L66ECMHDTYHAQ7VAVCNFSM6AAAAAA77NXUYKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZTGE3TQMBRGA. You are receiving this because you commented.Message ID: @.***>

small-zeng commented 10 months ago

是的，我是3090， 原来都不会增加，最近不知道总是超出

JIANG-CX commented 10 months ago

可能要回溯一下历史纪录看看哪里不同了hhh

获取 Outlook for iOShttps://aka.ms/o0ukef

发件人: Jing Zeng @.> 发送时间: Thursday, November 30, 2023 2:26:55 PM 收件人: SYSU-STAR/H2-Mapping @.> 抄送: JIANG Chenxing @.>; Comment @.> 主题: Re: [SYSU-STAR/H2-Mapping] loss曲线抖动幅度太大 (Issue #20)

是的，我是3090， 原来都不会增加，最近不知道总是超出

― Reply to this email directly, view it on GitHubhttps://github.com/SYSU-STAR/H2-Mapping/issues/20#issuecomment-1833180834, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APMKKNWVEM43AVMJ2CCTCRLYHARK7AVCNFSM6AAAAAA77NXUYKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZTGE4DAOBTGQ. You are receiving this because you commented.Message ID: @.***>

small-zeng commented 10 months ago

ok，我再试试，这个理论上是初始就分配好显存，还是给根据场景数据会逐渐变化呢

JIANG-CX commented 10 months ago

keyframe因为也存在cuda上，所以会逐渐增加

获取 Outlook for iOShttps://aka.ms/o0ukef

发件人: Jing Zeng @.> 发送时间: Thursday, November 30, 2023 2:39:19 PM 收件人: SYSU-STAR/H2-Mapping @.> 抄送: JIANG Chenxing @.>; Comment @.> 主题: Re: [SYSU-STAR/H2-Mapping] loss曲线抖动幅度太大 (Issue #20)

ok，我再试试，这个理论上是初始就分配好显存，还是给根据场景数据会逐渐变化呢

― Reply to this email directly, view it on GitHubhttps://github.com/SYSU-STAR/H2-Mapping/issues/20#issuecomment-1833192845, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APMKKNRLYCCCI5GC4RXKCB3YHASZPAVCNFSM6AAAAAA77NXUYKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZTGE4TEOBUGU. You are receiving this because you commented.Message ID: @.***>

small-zeng commented 9 months ago

请教个问题，如果我一开始拿到的位姿是有噪声的，但是后期由于SLAMF方法 全局BA 位姿变好了，重建质量可以变好吗，我看有个coarse tsdf， 这个会不会有影响

JIANG-CX commented 9 months ago

暂时没有加入ba后更新地图的功能

获取 Outlook for iOShttps://aka.ms/o0ukef

发件人: Jing Zeng @.> 发送时间: Thursday, December 14, 2023 1:57:40 PM 收件人: SYSU-STAR/H2-Mapping @.> 抄送: JIANG Chenxing @.>; Comment @.> 主题: Re: [SYSU-STAR/H2-Mapping] loss曲线抖动幅度太大 (Issue #20)

请教个问题，如果我一开始拿到的位姿是有噪声的，但是后期由于SLAMF方法 全局BA 位姿变好了，重建质量可以变好吗，我看有个coarse tsdf， 这个会不会有影响

― Reply to this email directly, view it on GitHubhttps://github.com/SYSU-STAR/H2-Mapping/issues/20#issuecomment-1855190581, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APMKKNW3OVP4S3RBCBHSTATYJKINJAVCNFSM6AAAAAA77NXUYKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJVGE4TANJYGE. You are receiving this because you commented.Message ID: @.***>