search

bagh2178 / SG-Nav

[NeurIPS 2024] SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigation
https://bagh2178.github.io/SG-Nav/
MIT License
52 stars 6 forks source link

Issues with environment setup #3

Open DanielAsadi opened 2 days ago

DanielAsadi commented 2 days ago

I've been trying for a while to get the SG-Nav environment setup but have had quite a bit of issues. I was wondering if I can get some advice.

Setup attempt when trying to follow SG_Nav.yml

Pip dependencies

  1. This version of tbb specified causes an issue: tbb=2020.2=h4bd325d_4 
    
...
Solving environment: failed

LibMambaUnsatisfiableError: Encountered problems while solving:

Could not solve for environment specs The following packages are incompatible ├─ numba ==0.53.1 py37ha9443f7_0 is installable and it requires │ └─ tbb >=2020.3 , which can be installed; └─ tbb ==2020.2 h4bd325d_4 is not installable because it conflicts with any installable versions previously reported.


2. After i do `tbb >=2020.3`, I get

... ERROR: Could not find a version that satisfies the requirement gradslam==1.0.0 (from versions: 0.1.0) ERROR: No matching distribution found for gradslam==1.0.0

To avoid this, I commented out this dependency and was able to install it successfully by cloning the repo and following the instructions in the readme: https://github.com/gradslam/gradslam.

3. After that I get

... ERROR: Could not find a version that satisfies the requirement groundingdino==0.1.0 (from versions: none) ERROR: No matching distribution found for groundingdino==0.1.0

To fix this, I found that the name of the package should actually be `groundingdino-py==0.1.0`

4. Next I get

... ERROR: Could not find a version that satisfies the requirement habitat==0.2.1 (from versions: 0.2.dev0, 0.2, 0.3, 0.3.1, 0.3.2, 0.3.3) ERROR: No matching distribution found for habitat==0.2.1

To avoid this, I comment out 
  - habitat==0.2.1
  - habitat-sim==0.2.1
and instead install it with conda from the instructions in https://github.com/facebookresearch/habitat-sim.

5. Then I get

... ERROR: Could not find a version that satisfies the requirement llava==1.1.3 (from versions: 0.0.1.dev0) ERROR: No matching distribution found for llava==1.1.3

To avoid this, instead of specifying `llava==1.1.3` I don't specify any version.

6. After that I get

... ERROR: Could not find a version that satisfies the requirement torch==1.9.1+cu111 (from versions: 1.0.0, 1.0.1, 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1) ERROR: No matching distribution found for torch==1.9.1+cu111


At this point I was quite frustrated and was not sure which torch/cuda versions to try so I stopped here and tried a different approach, trying to remove all the version specifications in the yml file.

## Setup attempt when removing all dependency versions and trying python3.9 instead
For this attempt, I removed all the locked dependency versions, still applied the same fixes that I mentioned above and then specified python to version 3.9. Overall, this seemed promising, but my main issue now is with the GLIP repo.

With python 3.9, I had to install `conda install pytorch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1  pytorch-cuda=11.8 -c pytorch -c nvidia` because that is the latest that `pytorch3d` supports, which is needed for the `concept-graphs` dependency. All the other pip dependencies install without issues.

### Habitat
I was able to install habitat sim without issues.

### GLIP
When trying to install GLIP, at first I was getting an error where the absence of a  `maskrcnn_benchmark` directory was causing a faliure of the setup.py script running. After creating a directory with that name,  I got this output:

... Installed /home/mfyuan/SG-Nav/GLIP Processing dependencies for maskrcnn-benchmark==0.0.0 Finished processing dependencies for maskrcnn-benchmark==0.0.0

I don't know why it installed "version" `0.0.0` so i specified the only version of `maskrcnn_benchmark` that exists, which was `0.1.0` and I got:

... Installed /home/mfyuan/SG-Nav/GLIP Processing dependencies for maskrcnn-benchmark==0.1.0 Finished processing dependencies for maskrcnn-benchmark==0.1.0

However, the main issue I am facing now is that while it seems the installation of `GLIP`/`maskrcnn_benchmark` is ok, it is not:

(sg-nav) mfyuan@mingfeng-TRAIL:~/SG-Nav$ python SG_Nav.py --evaluation local --reasoning both /home/mfyuan/SG-Nav/SG_Nav.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp Traceback (most recent call last): File "/home/mfyuan/SG-Nav/SG_Nav.py", line 30, in from GLIP.maskrcnn_benchmark.engine.predictor_glip import GLIPDemo ModuleNotFoundError: No module named 'GLIP.maskrcnn_benchmark.engine'


None of the `GLIP`/`maskrcnn_benchmark` modules imported are recognized, and I've been stuck here for a while.

Aditionally, I wasn't able to get much information or debugging on `GLIP` and `maskrcnn_benchmark`. While they say `pytorch>=1.9`, it doesn't seem to work. Is GLIP the reason for specifying the older pytorch/cuda version in the yml file? I saw this issue in their repo: https://github.com/microsoft/GLIP/issues/89#issuecomment-1884215904.

Also, `maskrcnn_benchmark` is 5 years old and is deprecated, so I wasn't able to figure out any ideas on how to address it either. In the `maskrcnn_benchmark` they mention to now use detectron2: https://github.com/facebookresearch/detectron2, but I am not sure how this would work with GLIP.

### Concept graphs
On the topic of the concept graphs repo, in the readme they specify to use the [`ali-dev` branch](https://github.com/concept-graphs/concept-graphs/tree/ali-dev) since the code was significantly updated compared to the main branch.

The setup for this has mostly been ok I believe, although I had to change some code since it uses some newer python 3.10/11 features.

**_Overall, I'd really appreciate if you can give some advice on how to approach the issues I mentioned above, and if you're able to try setting up the environment from scratch and update the yml file accordingly, to either use python3.7 or 3.9, whichever works at the end of the day is fine with my. I've spent a lot of time trying to get your intriguing research to work but have had no luck. Thanks again._**