search

Fangkang515 / CE3D

Official Implementation of ECCV2024 paper: Chat Edit 3D: Interactive 3D Scene Editing via Text Prompts
171 stars 5 forks source link

Requirement version #5

Closed YANGTUOMAO closed 1 month ago

YANGTUOMAO commented 1 month ago

Could you please tell me what kind of runtime environment I need to run the code smoothly? I encounter many dependency conflicts when using pip install -r requirements.txt. Could you provide a complete runtime environment to help determine which version numbers can be used? Thank you very much!

请问现在想顺利运行代码，需要什么样的运行环境？我使用pip install -r requirements.txt 会有很多依赖冲突。可不可以提供一下完整的运行环境，以方便确定可以使用的版本号？ 感激不尽！

Fangkang515 commented 1 month ago

I will check it today.

Fangkang515 commented 1 month ago

I'd like to confirm that the conflict you're experiencing is something like the following? img_v3_02ft_9efd98ee-f1f6-4763-8b5b-01a9a1bc533g

Fangkang515 commented 1 month ago

Hello, I updated the requirements.txt, please run 'git pull'. I install it on my device without any conflicts anymore. But be careful, please don't use Tsinghua's source for pip, it is missing a package. It is recommended to use aliyun source or directly scientifically access the Internet. 你好，我重新上传了 requirements.txt，请执行 git pull. 我自己安装测试已经没有任何冲突了。但需要注意，请不要使用清华的源进行pip,它缺少一个包。建议使用阿里的源或者直接科学上网~

YANGTUOMAO commented 1 month ago

Hello, I updated the requirements.txt, please run 'git pull'. I install it on my device without any conflicts anymore. But be careful, please don't use Tsinghua's source for pip, it is missing a package. It is recommended to use aliyun source or directly scientifically access the Internet. 你好，我重新上传了 requirements.txt，请执行 git pull. 我自己安装测试已经没有任何冲突了。但需要注意，请不要使用清华的源进行pip,它缺少一个包。建议使用阿里的源或者直接科学上网~

OK, i will try directly access the Internet. the conflict is right as u shown! thx 我再试试，万分感谢~

YANGTUOMAO commented 1 month ago

@Fangkang515 

(CE3D) yanzh@node28:~/project/CE3D$ python3 chat_edit_3D.py --port 7862 --clean_FBends --load "Segmenting_cuda:0,\
  ImageCaptioning_cuda:0,Text2Image_cuda:0,VisualQuestionAnswering_cuda:0,\
  Text2Box_cuda:0,Inpainting_cuda:0,InstructPix2Pix_cuda:0,Image2Depth_cuda:0,\
  DepthText2Image_cuda:0,SRImage_cuda:0,Image2Scribble_cuda:1,\
  ScribbleText2Image_cuda:1,Image2Canny_cuda:1,CannyText2Image_cuda:1,\
  Image2Line_cuda:1,LineText2Image_cuda:1,Image2Hed_cuda:1,HedText2Image_cuda:1,\
  Image2Pose_cuda:1,PoseText2Image_cuda:1,SegText2Image_cuda:1,\
  Image2Normal_cuda:1,NormalText2Image_cuda:1,ReferenceImageTextEditing_cuda:1"
Initializing VisualQuestionAnswering to cuda:0

......
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["bos_token_id"]` will be overriden.
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["eos_token_id"]` will be overriden.
Couldn't connect to the Hub: 404 Client Error. (Request ID: Root=1-671bc41a-332780821f9fd93e4ecf42be;8fa867ac-b600-48a0-a406-00089f55bd68)

Revision Not Found for url: https://huggingface.co/api/models/stable-diffusion-v1-5/stable-diffusion-inpainting/revision/fp16.
Invalid rev id: fp16.
Will try to load from local cache.
Traceback (most recent call last):
  File "/data1/yanzh/.conda/envs/CE3D/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 406, in hf_raise_for_status
    response.raise_for_status()
  File "/data1/yanzh/.local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/models/stable-diffusion-v1-5/stable-diffusion-inpainting/revision/fp16

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data1/yanzh/.conda/envs/CE3D/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 1656, in download
    info = model_info(pretrained_model_name, token=token, revision=revision)
  File "/data1/yanzh/.conda/envs/CE3D/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/data1/yanzh/.conda/envs/CE3D/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 2467, in model_info
    hf_raise_for_status(r)
  File "/data1/yanzh/.conda/envs/CE3D/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 413, in hf_raise_for_status
    raise _format(RevisionNotFoundError, message, response) from e
huggingface_hub.errors.RevisionNotFoundError: 404 Client Error. (Request ID: Root=1-671bc41a-332780821f9fd93e4ecf42be;8fa867ac-b600-48a0-a406-00089f55bd68)

Revision Not Found for url: https://huggingface.co/api/models/stable-diffusion-v1-5/stable-diffusion-inpainting/revision/fp16.
Invalid rev id: fp16

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data1/yanzh/project/CE3D/chat_edit_3D.py", line 3202, in <module>
    global_inpaint = Inpainting(device="cuda:0")
  File "/data1/yanzh/project/CE3D/chat_edit_3D.py", line 383, in __init__
    self.inpaint = StableDiffusionInpaintPipeline.from_pretrained(
  File "/data1/yanzh/.conda/envs/CE3D/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/data1/yanzh/.conda/envs/CE3D/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 1096, in from_pretrained
    cached_folder = cls.download(
  File "/data1/yanzh/.conda/envs/CE3D/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/data1/yanzh/.conda/envs/CE3D/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 1905, in download
    raise EnvironmentError(
OSError: Cannot load model runwayml/stable-diffusion-inpainting: model is not cached locally and an error occured while trying to fetch metadata from the Hub. Please check out the root cause in the stacktrace above.

Another error occurred. We have set up the environment using the requirements.txt, and some extra modules are installed that way. However, when it comes to running make run-all, the above error occurred. Do you have any suggestions?

我今晚重新创建了环境，在科学上网的保障下虽有坎坷但缝缝补补还是搭好了。但是为什么会有上述错误？看上去像是模型版本号找不到的问题？

Fangkang515 commented 1 month ago

我测试了下我这边是正常的结果： image

然后diffuser和transformer版本如下（和requirements.txt是一样的）： image

所以我觉得大概率还是huggingface访问的问题，因为我看到你报错里面有一个”Couldn't connect to the Hub: 404 Client Error.“

我建议先测试下面的example能否正常运行，如果下面的能运行的话，CE3D应该就没问题了： example1: https://huggingface.co/docs/diffusers/main/en/api/pipelines/controlnet#diffusers.StableDiffusionControlNetPipeline.enable_attention_slicing.example

example2: https://huggingface.co/docs/diffusers/using-diffusers/inpaint#stable-diffusion-inpainting

YANGTUOMAO commented 1 month ago

image May I ask if this is the final step? After an all-night download, this message appeared in my terminal. However, I still can't access the webpage on my personal computer even after setting up the SSH link. I just want to make sure if I've actually reached the final stage of development.

请问显示running on local URL: http://0.0.0.0:7862是意味着到了最后一步了吗？我现在正在尝试做

YANGTUOMAO commented 1 month ago

我测试了下我这边是正常的结果： image

然后diffuser和transformer版本如下（和requirements.txt是一样的）： image

所以我觉得大概率还是huggingface访问的问题，因为我看到你报错里面有一个”Couldn't connect to the Hub: 404 Client Error.“

我建议先测试下面的example能否正常运行，如果下面的能运行的话，CE3D应该就没问题了： example1: https://huggingface.co/docs/diffusers/main/en/api/pipelines/controlnet#diffusers.StableDiffusionControlNetPipeline.enable_attention_slicing.example

example2: https://huggingface.co/docs/diffusers/using-diffusers/inpaint#stable-diffusion-inpainting

我去这么巧，我今早刚到实验室，所以我是到最后一步了已经，是吗？

Fangkang515 commented 1 month ago

是的，你在本地打开这个7862端口的URL, 就可以尝试编辑了

YANGTUOMAO commented 1 month ago

是的，你在本地打开这个7862端口的URL, 就可以尝试编辑了

但还有一个错误没解决，在chat_edit_3D.py文件，第380行

class Inpainting:
    def __init__(self, device):
        self.device = device
        self.revision = "fp16" if "cuda" in self.device else None
        # self.revision = "main" if "cuda" in self.device else None 我只有改成main才能运行，否则会报上面的错

        self.torch_dtype = torch.float16 if "cuda" in self.device else torch.float32

        self.inpaint = StableDiffusionInpaintPipeline.from_pretrained(
            "runwayml/stable-diffusion-inpainting",
            revision=self.revision,
            torch_dtype=self.torch_dtype,
            safety_checker=StableDiffusionSafetyChecker.from_pretrained(
                "CompVis/stable-diffusion-safety-checker", torch_dtype=self.torch_dtype
            ),
        ).to(device)

如果我使用 self.revision = "fp16"会出现如下报错：

huggingface_hub.errors.RevisionNotFoundError: 404 Client Error. (Request ID: Root=1-671c6579-082eb3c0077e36ca7ffc5c19;54708d37-85cd-4628-96c9-b68e87442334)

Revision Not Found for url: https://huggingface.co/api/models/stable-diffusion-v1-5/stable-diffusion-inpainting/revision/fp16.
Invalid rev id: fp16

后面我选择改成 self.revision = "main"，就可以运行到最后的显示URL界面。 但是当我在本地web打开7862，输入./datasets/flower，会出现下面错误： RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1, 256, 256] because the unspecified dimension size -1 can be any value and is ambiguous 似乎无法加载图片，所以我不确定这个错误是不是和我改动了self.revision = "main" if "cuda" in self.device else None 相关 image

YANGTUOMAO commented 1 month ago

是的，你在本地打开这个7862端口的URL, 就可以尝试编辑了

我发了封邮件来描述更详细的信息，您有空看一下

Leonard-Yao commented 1 month ago

在huggingface中stable-diffusion-v1-5的模型界面已经是空的了，但是项目中提到的stable-diffusion-v2是可以用的，请问如何更新到v2版本模型

Fangkang515 commented 1 month ago

其实都可以的，我们的方法支持使用任何视觉模型。所以直接更改加载模型的代码就行，比如用你想用最新版的controlnet的话，就升级对应的diffuser和transformers库，然后跑通这里面的例子：https://huggingface.co/docs/diffusers/using-diffusers/controlnet.

最后将加载模型的那句直接替换chat_edit_3d.py对应的模型就行。

此外，虽然huggingface清空了，但可能还能继续用貌似。