search

cvachha / instruct-gs2gs

Gaussian splatting implementation of Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions
https://instruct-gs2gs.github.io/
MIT License
54 stars 0 forks source link

About dataset update interval #11

Open pansanity666 opened 3 months ago

pansanity666 commented 3 months ago

Hi,

A quick question is that if the total iteration is 30k, and the update interval is 2.5k, only (30/2.5)=12 images will be edited, which seems to be usually smaller than the dataset size. Is there something wrong with my understanding?

Best,

StephenYangjz commented 3 months ago

Hi @pansanity666, sry this seems to be a separate question: I wonder if you are able to get the most up to date code working? for me it runs without calling ip2p and doing the editing : /

pansanity666 commented 3 months ago

Hi @pansanity666, sry this seems to be a separate question: I wonder if you are able to get the most up to date code working? for me it runs without calling ip2p and doing the editing : /

I haven't tried to run this repo yet.

StephenYangjz commented 3 months ago

@pansanity666 Gotcha, if you do it would be greatly appreciated if you can lmk hows it going cuz im having this issue: https://github.com/cvachha/instruct-gs2gs/issues/10

cvachha commented 3 months ago

Hi @pansanity666, the full dataset of training images is edited/updated every 2.5k iterations. The pipeline in igs2gs_pipeline.py first edits all dataset images with Instruct-Pix2Pix, then trains the splat for 2.5k iterations, then repeats the process.

pansanity666 commented 3 months ago

Hi @pansanity666, the full dataset of training images is edited/updated every 2.5k iterations. The pipeline in igs2gs_pipeline.py first edits all dataset images with Instruct-Pix2Pix, then trains the splat for 2.5k iterations, then repeats the process.

Hi,

I understand your explanation but still kind of confused by the implementation code:

  1. It seems that you just edit 1 image every config.gs_steps here https://github.com/cvachha/instruct-gs2gs/blob/5890babbe169857e6764390e365f54ada8675783/igs2gs/igs2gs_pipeline.py#L115-L132

  2. Editing the whole dataset might be very time-consuming, making the training very slow?

Best,

cvachha commented 3 months ago

Hi, it iterates through all training images and edits one image at a time (also which ends up training 1 iteration per edited image), so the first [num of images] iterations are just for editing each individual image and training after replacing the original image with the edited one.

Once the full dataset has been updated (and it has trained for [num of images] iterations), we train for 2.5k (gs_steps) iterations on all the edited images, which applies the edit over the entire splat.

Since the edits are progressive (editing over the previously edited images after each dataset update), we generally recommend 2-3 dataset updates (training for about 5k-7.5k iterations). Editing the full dataset is a little time consuming (few seconds per training image) but applying the edits on the splat (training for 2.5k) is pretty quick.