Integrate Ha-Nerf in nerfstudio

[dberga]

First check in pipelines where "encode_a" and "use_mask" is applied

https://github.com/rover-xingyu/Ha-NeRF/blob/main/models/networks.py https://github.com/rover-xingyu/Ha-NeRF/blob/main/train_mask_grid_sample.py#L36 https://github.com/rover-xingyu/Ha-NeRF/blob/main/eval.py#L131 https://github.com/rover-xingyu/Ha-NeRF/blob/main/hallucinate.py#L178

Then re-implement a new case (take vanilla nerf as example)? Replacing the embeddings encoding and implicit mask https://github.com/dberga/nerfstudio/blob/main/nerfstudio/models/vanilla_nerf.py https://github.com/dberga/nerfstudio/blob/main/nerfstudio/fields/vanilla_nerf_field.py https://github.com/dberga/nerfstudio/blob/main/nerfstudio/field_components/embedding.py https://github.com/dberga/nerfstudio/blob/main/nerfstudio/field_components/encodings.py

After including the model in https://github.com/dberga/nerfstudio/blob/main/nerfstudio/models, you must include the model instruction name for "ns-train model" in https://github.com/dberga/nerfstudio/blob/main/nerfstudio/configs/method_configs.py

[dberga]

Some specific lines from the official repo

overall definitions (XingYu Ha-Nerf class)

https://github.com/rover-xingyu/Ha-NeRF/blob/main/train_mask_grid_sample.py#L36

forward usage (training)

https://github.com/rover-xingyu/Ha-NeRF/blob/main/train_mask_grid_sample.py#L121 https://github.com/rover-xingyu/Ha-NeRF/blob/main/train_mask_grid_sample.py#L128

inference usage (eval)

https://github.com/rover-xingyu/Ha-NeRF/blob/main/eval.py#L147

rendering inference with appearance encoding

https://github.com/rover-xingyu/Ha-NeRF/blob/main/models/rendering.py#L112

[dberga]

Added Ha-Nerf model definitions and imported main modules from official repo https://github.com/dberga/nerfstudio/commit/283511a2ecfe4101177a8c7409c8896d821eaf17 Taken Vanilla Nerf definition and added appearance field (using E_attr) and occlusion (implicit_mask).

See ha-nerf definition using vanilla-nerf structure and added ha-nerf fields / modules: https://github.com/dberga/nerfstudio/blob/main/nerfstudio/models/ha_nerf.py See imported functions here: https://github.com/dberga/nerfstudio/blob/main/nerfstudio/field_components/hallucidated.py See field definitions here: https://github.com/dberga/nerfstudio/blob/main/nerfstudio/fields/ha_nerf_field.py See execution config (and default params, defined keyword for ns-train as "ha-nerf"): https://github.com/dberga/nerfstudio/blob/main/nerfstudio/configs/method_configs.py

Pending / to do: -Remake field definitions (./nerfstudio/fields/ha_nerf_field.py) -Update ./nerfstudio/models/ha_nerf.py usage