search

JeffWang987 / OpenOccupancy

[ICCV 2023] OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception
Apache License 2.0
551 stars 47 forks source link

Apply single-GPU debug through pycharm #39

Open onionysy opened 9 months ago

onionysy commented 9 months ago

We want to debug using a single graphics card with pycharm, instead of using distributed training. But we ran into the following problems:

fatal: not a git repository (or any of the parent directories): .git 2023-09-26 21:37:31,804 - mmdet - INFO - Environment info: sys.platform: linux Python: 3.8.17 (default, Jul 5 2023, 21:04:15) [GCC 11.2.0] CUDA available: True GPU 0: NVIDIA GeForce RTX 3090 CUDA_HOME: /usr/local/cuda NVCC: Build cuda_11.3.r11.3/compiler.29920130_0 GCC: gcc (GCC) 6.1.0 PyTorch: 1.10.1 PyTorch compiling details: PyTorch built with:

We thought this was a SyncBN problem, so we changed it to BN in the configuration file, but encountered the following problems:(It is worth noting that while we are still using sysnBN, it is possible to debug through tesy.py)

fatal: not a git repository (or any of the parent directories): .git 2023-09-26 21:27:49,543 - mmdet - INFO - Environment info: sys.platform: linux Python: 3.8.17 (default, Jul 5 2023, 21:04:15) [GCC 11.2.0] CUDA available: True GPU 0: NVIDIA GeForce RTX 3090 CUDA_HOME: /usr/local/cuda NVCC: Build cuda_11.3.r11.3/compiler.29920130_0 GCC: gcc (GCC) 6.1.0 PyTorch: 1.10.1 PyTorch compiling details: PyTorch built with:

We know that this is a dimensional error, but there is no idea about this problem. Do you have any good suggestions?

JeffWang987 commented 9 months ago

You should change the BN configurations at here (SyncBN-->BN)

Agito555 commented 2 weeks ago

You should change the BN configurations at here (SyncBN-->BN)

Change the config as you advise and then encounter the same problem ''ValueError: expected 4D input (got 2D input)'' as mentioned aboved

Agito555 commented 2 weeks ago

Modiffying the code (SyncBN --> BN) doesn't work for me, which will raise error ValueError: expected 4D input (got 2D input) Actually, there is no need to modify the code (SyncBN --> BN) for debugging within a single GPU. Here is the solution for debugging in Vscode, I guess there is a similar solution to Pycharm.

You can add the following content into your launch.json and press F5 to start debugging.

{
            "name": "Python Distributed_training",
            "type": "debugpy",
            "request": "launch",
            "module": "torch.distributed.launch",
            "console": "internalConsole",
            "cwd": path/to/your/own/project/dir,
            "justMyCode": false,
            "env": {
                "CUDA_VISIBLE_DEVICES": "0"
            },
            "args": [
                "--nnodes", "1",
                "--node_rank","0",
                "--nproc_per_node", "1",
                "--master_addr", "127.0.0.1",
                "--master_port", "29501",
                "./tools/train.py",
                path/to/your/config
                "--seed","0",
                "--launcher","pytorch"

            ]
        },

It works for me, hope it may help somebody who meets the same problem. Reference1 Reference2 Reference3