search

Ruyi-Zha / endosurf

MIT License
47 stars 3 forks source link

Generate .pkl file from poses_bounds.npy of EndoNeRF #9

Open smoreira00 opened 9 months ago

smoreira00 commented 9 months ago

I used several private datasets to run EndoNeRF and everything went well. Now I'm trying to use the same datasets to train EndoSurf, generating the .pkl files using the poses_bounds.npy files previously obtained, but the final results aren't correct. I think the problem may come from the .pkl file, but I don't know where since the EndoNeRF worked well for those same poses_bounds.npy files.

Do you have suggestion to solve my problem? Should I try to generate the .pkl files by myself and not from the poses_bounds.npy files?

Ruyi-Zha commented 8 months ago

Hi, thanks for your interest. preprocess.py is used to transform the scene into a unit sphere. You can add --show to visualize the camera trajectory and shape, checking if there is something wrong. My suggestion is to double-check the pose, and the scale of depth. Note that the camera of endonerf dataset is still. If your camera is moving, you might need to refer to scared2019/preprocess.py for some guidance.

smoreira00 commented 8 months ago

Thank you so much for your answer! Yes, my camera is moving and I was using the endonerf/preprocess.py file, so I may need to use the scared2019/preprocess.py. I appreciate the suggestion!

smoreira00 commented 8 months ago

In scared2019/preprocess.py why did you do this:

if i_frame == 0: c2w0 = c2w c2w = np.linalg.inv(c2w0) @ c2w

?

Ruyi-Zha commented 8 months ago

We set the first camera pose as the coordinate center, and transform all frames accordingly. Since we will then transform them into a unit sphere, ignoring this code should be fine.

smoreira00 commented 8 months ago

Hi, thanks for your interest. preprocess.py is used to transform the scene into a unit sphere. You can add --show to visualize the camera trajectory and shape, checking if there is something wrong. My suggestion is to double-check the pose, and the scale of depth. Note that the camera of endonerf dataset is still. If your camera is moving, you might need to refer to scared2019/preprocess.py for some guidance.

In my case, the camera's moving. Is there much of a difference between a still and a moving camera that justifies having to refer to scared2019/preprocess.py instead of endonerf/preprocces.py to generate the .pkl file?

Ruyi-Zha commented 8 months ago

There are not many differences between the two preprocess.py except for some data reading stuff. However, since the endonerf camera is static, I did not debug if endonerf/preprocess.py works for moving cameras.

I think the easiest way to check if the generated .pkl file is correct is to visualize it with --show. Please check 1) if the trajectory (pose) is correct, 2) if camera frustums (intrinsics) look ok, 3) if the scene is inside the unit sphere. If everything looks good, the .pkl file should be fine. In this case, you might need to tune training hyperparameters.

It would be good if you could share with me some sample data and scene visualization so that I can give you some ideas. Thanks.

smoreira00 commented 8 months ago

Thank you for your help!

I already checked some stuff, but I'll check again. I just have two questions:

  1. Do the bounds (bds) that come from the poses_bounds.npy file have a big influence on the EndoSurf's training? Where and how do they enter the model?
  2. Is there any problem if the cameras' origins are placed inside the unit sphere?