search

Genesis-Embodied-AI / RoboGen

A generative and self-guided robotic agent that endlessly propose and master new skills.
Apache License 2.0
552 stars 50 forks source link

error when downloading new object #11

Closed quantumiracle closed 9 months ago

quantumiracle commented 9 months ago

Hi,

Thanks for the great contribution!

I found the error when it tries to execute a task with a new object, e.g. water container, the error appear in the object downloading process:

========== candidate 0 similarity: 0.861998438835144 ==========
found uid:  4f88fa8855c6428d8d57887694f8b0d8
found description:  Water Container / Carrier
...
FileNotFoundError: [Errno 2] No such file or directory: 'objaverse_utils/data/images/4f88fa8855c6428d8d57887694f8b0d8.jpeg'

Any help on this would be appreciated!

Zihan

yufeiwang63 commented 9 months ago

Hi,

I think this is probably because the path objaverse_utils/data/images does not exist. For a temporary workaround you can manually create this folder. I will fix this for downloading new objects.

quantumiracle commented 9 months ago

Thanks for your reply.

However after I added this folder, another missing file error appears:

with open(save_path, "w") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'data/debug/verify.json'
yufeiwang63 commented 9 months ago

Can you provide a bit more detail here? What command did you run, and what is the full error message?

Thanks

quantumiracle commented 9 months ago

I think the problem is caused by a rejection of an object to download:

==================== trying to download object: a water container ====================
INFO - 2023-11-24 15:57:55,576 - blip2_t5 - freeze vision encoder
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████| 2/2 [00:07<00:00,  3.50s/it]
INFO - 2023-11-24 15:58:22,172 - blip2 - load checkpoint from[ https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/BLIP2/blip2_pretrained_flant5xl.pth](https://wx.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=https%3A%2F%2Fstorage.googleapis.com%2Fsfr-vision-language-research%2FLAVIS%2Fmodels%2FBLIP2%2Fblip2_pretrained_flant5xl.pth&skey=%40crypt_b4874f0d_98c35bc85394d92fcb66d57f2ff5eeda&deviceid=e772055577453250&pass_ticket=undefined&opcode=2&scene=1&username=@aa408d1a8bf8d0427bcd6649d1303eb13f09d5982d84875dd60d2f5ca7728c89)
searching whole objaverse for:  a water container
INFO - 2023-11-24 15:59:53,302 - SentenceTransformer - Load pretrained SentenceTransformer: all-mpnet-base-v2
INFO - 2023-11-24 15:59:54,357 - SentenceTransformer - Use pytorch device: cuda
Batches: 100%|█████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.65it/s]
...

========== candidate 0 similarity: 0.861998438835144 ==========
found uid:  4f88fa8855c6428d8d57887694f8b0d8
found description:  Water Container / Carrier

            A robotic arm is trying to solve a task to learn a manipulation skill in a simulator.
        We are trying to find the best objects to load into the simulator to build this task for the robot to learn the skill.
        The task the robot is trying to learn is: Poor water into a cup. 
        A more detailed description of the task is: The robot arm will pick up a water container, open the lid of the cup, pour water into the cup, close the lid, and then place the water container back to its original position.
        As noted, to build the task in the simulator, we need to find this object: a water container.
        We are retrieving the object from an existing database, which provides some language annotations for the object.
        With the given lanugage annotation, please think if the object can be used in the simulator as a water container for learning the task Poor water into a cup.

        This is the language annotation:
        a 3d model of a white object on a grey background

        Please reply first with your reasoning, and then a single line with "**yes**" or "**no**" to indicate whether this object can be used.

=====================================
Based on the language annotation provided, it is not possible to determine if the object can be used as a water container. The description "a 3d model of a white object on a grey background" is too vague and does not provide any specific details about the object's shape, size, or functionality, which are crucial to determine if it can serve as a water container.

**No**

The above error appears after this, but this rejection seems to be the problem.

It's in generating a task like 'pour water in to a cup' with object 'cup'

yufeiwang63 commented 9 months ago

Sorry for the late reply here. I have made a new commit that should fix both of the missing directories:

https://github.com/Genesis-Embodied-AI/RoboGen/blob/main/gpt_4/bard_verify.py#L69-L70 https://github.com/Genesis-Embodied-AI/RoboGen/blob/main/gpt_4/bard_verify.py#L103-L104

Let me know if you still face the issue.

BTW, the pouring water task might not work with pybullet since it does not really support fluid simulation (a workaround is to simulate water as a collection of small rigid spheres, but the current repo does not implement it yet. )