Render with edit error: RuntimeError: mat1 and mat2 shapes cannot be multiplied (105562x256 and 512x1)

[zxczhai]

i follow the step and use the extraction banana's dataset,but i have the wrong as follow. Could you tell me how to solve it?Thanks a lot!

(feature-3dgs) PS F:\gaussian_splatting\feature-3dgs> python render.py -s F:\gaussian_splatting\dataset\feature3dGS\sample_dataset -m output/fruits -f sam --iteration 7000 --edit_config configs/edit_extraction.yaml
Looking for config file in output/fruits\cfg_args
Config file found: output/fruits\cfg_args
Rendering output/fruits
Loading trained model at iteration 7000 [25/02 14:38:41]
Reading camera 57/57 [25/02 14:38:41]
Loading Training Cameras [25/02 14:38:41]
Loading Test Cameras [25/02 14:38:44]
{'edit': {'objects': ['fruit'], 'operations': 'extraction', 'targets': 'fruit', 'threshold': None}} [25/02 14:38:45]
Rendering progress:   0%|                                                                                                                                                                    | 0/57 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "render.py", line 389, in <module>
    render_sets(model.extract(args), args.iteration, pipeline.extract(args), args.skip_train, args.skip_test, args.novel_view,
  File "render.py", line 353, in render_sets
    render_set(dataset.model_path, "train", scene.loaded_iter, scene.getTrainCameras(), gaussians, pipeline, background, edit_config, dataset.speedup)
  File "render.py", line 128, in render_set
    render_pkg = render_edit(view, gaussians, pipeline, background, text_feature, edit_dict)
  File "F:\gaussian_splatting\feature-3dgs\gaussian_renderer\__init__.py", line 142, in render_edit
    scores = calculate_selection_score(semantic_feature[:, 0, :], text_feature,
  File "F:\gaussian_splatting\feature-3dgs\gaussian_renderer\__init__.py", line 24, in calculate_selection_score
    scores = features.half() @ query_features.T.half()  # (N_points, n_texts)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (105562x256 and 512x1)

edit_extraction.yaml

edit:
  objects: ["fruit"]
  operations: "extraction" 
  targets: "fruit" # the target object should be chosen from the object list
  threshold: 0.24 # leave it empty if using argmax

when i change "fruit" to "apple","banana".it wrong either.

[41xu]

I have the same questions👀🤔 Have you figured it out? @zxczhai

[beautifulchoi]

I have the same questions👀🤔 Have you figured it out? @zxczhai

Did you use speedup while training? it seems that because of CLIP text encoder, the size of feature dimension should be same. so I think each gaussian dim should be 512 when doing downsteam task with lseg

[41xu]

I have the same questions👀🤔 Have you figured it out? @zxczhai

Did you use speedup while training? it seems that because of CLIP text encoder, the size of feature dimension should be same. so I think each gaussian dim should be 512 when doing downsteam task with lseg

That's indeed a prior that the query feature dim (which is actually the clip dim) and the generated feature dim must be the same. So if we use speedup for training, it cannot perform the query, right?

[ShijieZhou-UCLA]

Hi there! Yes, for the language-guided editing task, since the editing happened in 3D, we have to make the feature dimension on each 3D Gaussian match with the CLIP text feature dimension, which is 512. Therefore, as indicated in our readme, we don't use --speedup for this task.