Is it possible to input more than five views?

gerwang commented 2 years ago

Hi, thanks for the great work!

If I want to input more than five views (for example, to fairly compare with works like NeRFactor), is it possible? Should I expect the results to be better than inputting five views?

ken2576 commented 2 years ago

Hi, Yes it should be possible. You could take a look at https://github.com/ken2576/nelf/blob/main/run_validate_view.py We change the view count to show different rendering quality. I think you could prepare data for more views and compare with other methods. For more than 5 views, I think the specularity should be handled better since there are more views on the high-frequency highlights. However, we did not test the algorithm in that situation, so let me know if it is acting otherwise.

gerwang commented 2 years ago

I can feed 31 views to the network now. But I find the results are worse than inputting five views. Here are my result videos:

I found more "black holes" in the generated geometry, which caused artifacts.

Here is how I obtained the results:

The processed input can be found here: https://drive.google.com/file/d/1O9tj9emonnTlKNkQy8dF_oHBZrorFwWG/view?usp=sharing. I unzipped the file, and put the directory source_image_masked into ./data_test/processed_input.

I modified run_test.py to test on the processed input with 31 views:

##### Test #####
if True or os.sys.argv[2].startswith('validate'):
    light = cv2.imread(f'{arg.base_path}/data_test/{os.sys.argv[2]}/source_image_masked/target_light.hdr', cv2.IMREAD_UNCHANGED)
    light = cv2.resize(
        cv2.cvtColor(light, cv2.COLOR_BGR2RGB), tuple(light_size[::-1]), interpolation=cv2.INTER_AREA
    )
    cam_poses = [(15, 22.5*i) for i in range(16)] + [(15, 0) for i in range(32)]
    lights = [0 for i in range(16)] + [15-i for i in range(16)] + [np.roll(light, i, axis=1) for i in range(16)]
    filenames = [f'{i}' for i in range(48)]
    tester(
        data_name=os.sys.argv[2], image_ids=list(range(31)), # for 31 views,  [0, 22, 12, 14, 3] for five views
        dataset_color=None, extra_color_scale=1.0,
        cam_poses=cam_poses,
        lights=lights,
        filenames=filenames,
        model=model, test_path=f'{arg.base_path}/data_test', folder='masked'
    )

I start the program with python run_test.py nelf_ft processed_input 500000

Could you please tell me how to remove these artifacts? Where have I gone wrong? Thank you!

ken2576 commented 2 years ago

Sorry for the late reply. Since our network was trained with only 5 views as input, you might need to retrain it first. Another thing is that the data seems a bit too close. Could you try to make sure the full face is visible in the image?

ken2576 / nelf

Is it possible to input more than five views? #5