search

lukemelas / realfusion

Official code for "RealFusion: 360° Reconstruction of Any Object from a Single Image" (CVPR 2023)
Apache License 2.0
543 stars 38 forks source link

janus problem #19

Open MnKnight1 opened 1 year ago

MnKnight1 commented 1 year ago

Thank you for your work! I found that there was a janus problem in the result of the "teddy bear" example, because the textural inversion was over-fitted to its front view, resulting in not producing the correct rear view, which could lead to janus problems. Will there be such a phenomenon in the official result?

https://github.com/lukemelas/realfusion/assets/78358378/43055208-c1f3-4a98-913e-e08d7524cb8e

My command is as follows:

export MODEL_NAME="/home/litaiqing/.cache/huggingface/hub/models--runwayml--stable-diffusion-v1-5/snapshots/aa9ba505e1973ae5cd05f5aedd345178f52f8e6a"
export DATA_DIR="/media/ssd_1/litaiqing/realfusion-main/examples/natural-images/teddy_bear_1"
export OUTPUT_DIR="/media/ssd_1/litaiqing/realfusion-main/examples/natural-images/teddy_bear_1"

CUDA_VISIBLE_DEVICES=7 python textual_inversion.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --train_data_dir=$DATA_DIR \
  --learnable_property="object" \
  --placeholder_token="_teddy_bear_" \
  --initializer_token="teddy " \
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --max_train_steps=3000 \
  --learning_rate=5.0e-04 --scale_lr \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --output_dir=$OUTPUT_DIR \
  --use_augmentations

export DATA_DIR=/media/ssd_1/litaiqing/realfusion-main/examples/natural-images/teddy_bear_1

CUDA_VISIBLE_DEVICES=7 python main.py --O \
    --image_path $DATA_DIR/rgba.png \
    --learned_embeds_path $DATA_DIR/learned_embeds.bin \
    --text "a  _teddy_bear_" \
    --pretrained_model_name_or_path "/home/litaiqing/.cache/huggingface/hub/models--runwayml--stable-diffusion-v1-5/snapshots/aa9ba505e1973ae5cd05f5aedd345178f52f8e6a"