search

prs-eth / PCAccumulation

[ECCV 2022] Dynamic 3D Scene Analysis by Point Cloud Accumulation
https://shengyuh.github.io/pcaccumulation/index.html
MIT License
120 stars 10 forks source link

Memory Overflow Issue #8

Open plpalmer opened 1 year ago

plpalmer commented 1 year ago

Hello,

i have noticed that used system memory increased over the course of the training. The training of the model than crashes when the system memory is completely filled up. It usually takes multiple Epochs. Have you noticed this issue before? Do you have an explanation, or some idea what could cause this issue and what i should look further into? I use the model with my own data, but have also noticed this issue when used with your provided Waymo data.

Thanks for your help in advance

plpalmer commented 1 year ago

I think that i have found the problem. In models/alignnet.py im not sure what the parameter updated_inst_motion does. It does not seem to get used at any other point after line 254. When i comment out line 250-254 it does not seem to influence training but i dont have the memory overflow issue. It seems like updated_inst_motion is multiplied with c_inst_pose_est but because it never gets used in a backpropagation step it does not get cleared out from the computational graph. Could you elaborate on what the updated_inst_motion describes exactly and why it is set that way?

EasonChaozhou commented 6 months ago

Hello author, how can I visualize and achieve the qualitative results like in figure 8 in the paper?