Editing the scene

[PeterNg15]

Hi @LightwheelAI

On the website, the editing results are static, I was wondering if you can edit/add moving assets to the scene as well?

Finally, could you provide us with a script to edit the scene? That would greatly be appreciated. Thanks

Best, Peter

[LightwheelAI]

Hi, thanks for your attention!

Since we have achieved decompositing of dynamic objects from static backgrounds, replacement and addition of dynamic targets is possible. This can be achieved by editing the bbox and the parameters of the object model in ckpt. As we are currently working on other projects, we are unable to provide the corresponding scripts at this time. We look forward to providing more efficient tools for you to use in the future!

Best

[PeterNg15]

Hi,

Thanks for your reply. Would it not be possible for you to share the script you used for editing the scene in figure 8 of the paper? I imagine that it is already available

[LightwheelAI]

Hi, you can edit the value in annotation.json to change dynamic objects' pose and use the follow script to change specified object. best

import os
import shutil
import torch

def change_object(ckpt1, ckpt2, gid1, gid2, save_path):
    obj_key1 = f'object_{gid1}'
    obj_key2 = f'object_{gid2}'

    obj1_attr_list = [attr for attr in ckpt1['pipeline'].keys() if obj_key1 in attr]
    for attr in obj1_attr_list:
        attr2 = attr.replace(obj_key1, obj_key2)
        if attr2 not in ckpt2['pipeline'].keys():
            continue
        ckpt1['pipeline'][attr] = ckpt2['pipeline'][attr2]

    torch.save(ckpt1, save_path)

if __name__ == '__main__':
    root1 = '/path/to/scene1'
    root2 = '/path/to/scene2'

    ckpt1 = torch.load(f'{root1}/nerfstudio_models/step-000069999.ckpt')
    ckpt2 = torch.load(f'{root2}/nerfstudio_models/step-000069999.ckpt')
    gid1 = 'source_object_gid'
    gid2 = 'target_object_gid'
    save_folder = '/path/to/save/new/scene'
    if not os.path.exists(save_folder):
        os.mkdir(save_folder)
        os.mkdir(f'{save_folder}/nerfstudio_models')
    if not os.path.exists(f'{save_folder}/config.yml'):
        shutil.copy(f'{root1}/config.yml', save_folder)
    save_path = f'{save_folder}/nerfstudio_models/step-000069999.ckpt'
    change_object(ckpt1, ckpt2, gid1, gid2, save_path)

[m15310926778]

Hi, you can edit the value in annotation.json to change dynamic objects' pose and use the follow script to change specified object. best

import os
import shutil
import torch

def change_object(ckpt1, ckpt2, gid1, gid2, save_path):
    obj_key1 = f'object_{gid1}'
    obj_key2 = f'object_{gid2}'

    obj1_attr_list = [attr for attr in ckpt1['pipeline'].keys() if obj_key1 in attr]
    for attr in obj1_attr_list:
        attr2 = attr.replace(obj_key1, obj_key2)
        if attr2 not in ckpt2['pipeline'].keys():
            continue
        ckpt1['pipeline'][attr] = ckpt2['pipeline'][attr2]

    torch.save(ckpt1, save_path)

if __name__ == '__main__':
    root1 = '/path/to/scene1'
    root2 = '/path/to/scene2'

    ckpt1 = torch.load(f'{root1}/nerfstudio_models/step-000069999.ckpt')
    ckpt2 = torch.load(f'{root2}/nerfstudio_models/step-000069999.ckpt')
    gid1 = 'source_object_gid'
    gid2 = 'target_object_gid'
    save_folder = '/path/to/save/new/scene'
    if not os.path.exists(save_folder):
        os.mkdir(save_folder)
        os.mkdir(f'{save_folder}/nerfstudio_models')
    if not os.path.exists(f'{save_folder}/config.yml'):
        shutil.copy(f'{root1}/config.yml', save_folder)
    save_path = f'{save_folder}/nerfstudio_models/step-000069999.ckpt'
    change_object(ckpt1, ckpt2, gid1, gid2, save_path)

Do I need to retrain after changing annotation.json, and how to use this script specifically. thank you