search

ActiveVisionLab / DFNet

DFNet: Enhance Absolute Pose Regression with Direct Feature Matching (ECCV 2022)
https://dfnet.active.vision
MIT License
95 stars 9 forks source link

Question about rendering of the pre-trained nerfh model #10

Closed Tausc closed 11 months ago

Tausc commented 1 year ago

Hello! Thank you for your contribution! I've loaded your pre-trained nerfH model, but i meet some problem when rendering new image with it: 0

while the real-world image looks like: frame-000000 color

Do you know why this situation is happening?

chenusc11 commented 1 year ago

Hi, I think this looks like this file has not been properly load. DFNet/data/7Scenes/chess/pose_avg_stats.txt

Could you please check if you have put the pose_avg_stats.txt into the 7scene folder?

Tausc commented 1 year ago

Thank you for replying! So I add the pose_avg_stats.txt and function fix_coord() into my rendering code, but i still get some weird rendered images. I read relative code using pose_avg_stats.txt. It seems like this file is used to adjust the input pose. How could it improve the rendered images?

chenusc11 commented 1 year ago

If I remember this correctly, this is due to the position encoding of the NeRF. Since the query positions of each ray are embedded by the Fourier function (P.E. like Transformer, see NeRF paper), the input xyz values have to be within +- pi. fix_coord() ensures all query ray positions are within this range.

chenusc11 commented 1 year ago

Sorry. I misread your question. To improve the rendered image, you might use a better "ground truth pose" other than the dataset provided ones. Say COLMAP. The reason is the current GT poses are provided by depth-based slam, which in nature is a bit off-sync with the RGB images. Therefore, the rendering quality is not as good.

Tausc commented 1 year ago

So the poor rendering results is caused because the input pose and histogram do not match when calling the rendering function? Is there any problem with this understanding?

chenusc11 commented 1 year ago

Hi, really sorry for the late reply. I was on a leave due to family issue.

Have you solved your problem? If not, we could follow up your problem via email. I'm not exactly sure what you mean by “weird rendered images”. Maybe send me some of your examples?

chenusc11 commented 11 months ago

Close the issue for now. Feel free to re-open it if there's further question