dataset

[mengzhaoZ]

Hello, I would like to know how to process my own data set. The medical data set in the video of my data set moves slowly, the data I processed cannot meet the data requirements in the paper, and the training results are also poor. May I ask what I need to pay attention to most when processing other video data

[wangrun20]

In my series of experiments, the face selfie videos perform the best and most stable.

[mengzhaoZ]

In my series of experiments, the face selfie videos perform the best and most stable.

Thank you for your reply，my data set is the endoscope data set, I don't know whether it is applicable, and is there any requirement for FPS in video segmentation? My data camera moves slowly, will it affect the quality of training？

[wangrun20]

When preparing datasets for HyperNeRF, it captures some frames (default: 100 frames) from the video at equal intervals, then reconstructs on the basis of these frames. So I think the key to good reconstruction performance is not high frame rate or camera moving speed, but enough pictures from different observation angles.

[mengzhaoZ]

Thank you for your answer. I see what you mean. I think you're right. My data is different from the face data. It is not a relatively fixed rotating object, but a video image taken in the abdominal cavity. Maybe my video is not suitable for this method, so I will find another solution to see if it can be solved. Thanks again for your reply！

[wangrun20]

Thank you for your answer. I see what you mean. I think you're right. My data is different from the face data. It is not a relatively fixed rotating object, but a video image taken in the abdominal cavity. Maybe my video is not suitable for this method, so I will find another solution to see if it can be solved. Thanks again for your reply！

You're welcome. Wish you success!

[Zvyozdo4ka]

In my series of experiments, the face selfie videos perform the best and most stable.

I tried both selfie, and back camera experiments. In both cases the performance is disappointing

Is there a specific requirement how to take video?

https://github.com/google/hypernerf/assets/74532816/4e8a10f1-9a63-433a-b048-72799bff6228

https://github.com/google/hypernerf/assets/74532816/ea78a5db-c33c-468a-b814-f6e6e52fd262

[wangrun20]

I tried both selfie, and back camera experiments. In both cases the performance is disappointing

I said "In my series of experiments, the face selfie videos perform the best and most stable". Actually, what I meant is that the results for portrait selfies are slightly better than for object motion (like pouring water or moving things), not that the portrait selfies are extremely realistic. My reconstructing performance of portrait selfies is the same as your two videos. I think that even now, reconstructing 3D structures from a monocular camera is still very challenging, and I suspect that the authors of the HyperNeRF paper likely used some tricks that were not explicitly mentioned in the paper.

I think it is still very difficult to achieve the performance claimed by the author on our own dataset. It requires some tricks that I am not aware of.

[Zvyozdo4ka]

@wangrun20 thank you very much for your explanation!

[doubi-killer]

In my series of experiments, the face selfie videos perform the best and most stable.

I tried both selfie, and back camera experiments. In both cases the performance is disappointing

Is there a specific requirement how to take video?

output_hypernerf4_v1.mp4 output.mp4

how can i prepare a face selfie dataset? the colmap didn't run properly

[Zvyozdo4ka]

how can i prepare a face selfie dataset? the colmap didn't run properly

i tried their suggested colab code to preprocess video, but there were some issues to run it, i changed it a little bit to make it work. Here is their solution: https://colab.research.google.com/github/google/nerfies/blob/main/notebooks/Nerfies_Capture_Processing.ipynb

Here is how i did it, almost same, but as i remember there were the problem with colmap: https://colab.research.google.com/drive/1wZolsOwdYyo1xVli1cCS1bkC0JeII-CF?usp=sharing

As i remember, it must be --output_path in reconstruction module, instead of --export_path