Report a bug: warp?(codes can be run well in my workstation)

[leung1010]

when I excute the command: xxx$python experiments/scripts/dataset/main.py I got this, as follow: `env: blob: bsdf_pcd: type: diffuse reflectance: type: rgb value:

• 0.92941176
• 0.32941176
• 0.23137255 material: elasticity: cls: CorotatedElasticity E: 100000.0 nu: 0.2 requires_grad: false random: false plasticity: cls: IdentityPlasticity requires_grad: false random: false name: jelly ckpt: null shape: type: cube name: dataset center:
  • 0.5
  • 0.5
  • 0.5 size:
  • 0.5
  • 0.5
  • 0.5 resolution: 10 mode: uniform sort: null vel: random: false lin_vel:
  • 1.0
  • -1.5
  • -2.0 ang_vel:
  • 4.0
  • 4.0
  • 4.0 name: jelly rho: 1000.0 span:
    • 0
    • 1000 clip_bound: 0.5 render: spp: 32 width: 512 height: 512 skip_frame: 25 bound: 1.75 mpm_mul: 6 sph_version: cuda_ad_rgb pcd_version: cuda_ad_rgb has_sphere_emitter: true fps: 10 sim: quality: low num_steps: 1000 gravity:
    • 0.0
    • -9.8
    • 0.0 bc: freeslip num_grids: 20 dt: 0.0005 bound: 3 eps: 1.0e-07 skip_frame: 1 train: teacher: strategy: cosine start_lambda: 25 end_lambda: 200 num_epochs: 300 batch_size: 128 elasticity_lr: 1.0 plasticity_lr: 0.1 elasticity_wd: 0.0 plasticity_wd: 0.0 elasticity_grad_max_norm: 0.1 plasticity_grad_max_norm: 0.1 name: jelly/dataset seed: 0 cpu: 0 num_cpus: 128 gpu: 0 overwrite: false resume: false gt_name: null dataset: true

Warp 0.6.1 initialized: CUDA not available Devices: "cpu" | x86_64 Kernel cache: /home/xxxx/.cache/warp/0.6.1 Error executing job with overrides: ['overwrite=False', 'resume=False', 'gpu=0', 'cpu=0', 'env=jelly', 'render=debug', 'sim=low', 'name=jelly/dataset', 'dataset=True'] Traceback (most recent call last): File "/home/xxxx/xxxx_research/NCLaw/experiments/eval.py", line 137, in main() File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main _run_hydra( File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra _run_app( File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app run_and_report( File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report raise ex File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report return func() File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in lambda: hydra.run( File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/internal/hydra.py", line 132, in run = ret.return_value File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value raise self._return_value File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job ret.return_value = task_function(task_cfg) File "/home/xxxx/xxxx_research/NCLaw/experiments/eval.py", line 35, in main wp_device = wp.get_device(f'cuda:{cfg.gpu}') File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/warp/context.py", line 1676, in get_device return runtime.get_device(ident) File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/warp/context.py", line 1484, in get_device return self.device_map[ident] KeyError: 'cuda:0' env: blob: bsdf_pcd: type: diffuse reflectance: type: rgb value:

• 0.96078431
• 0.88235294
• 0.63529412 material: elasticity: cls: SigmaElasticity E: 1000000.0 nu: 0.2 requires_grad: false random: false plasticity: cls: DruckerPragerPlasticity E: 1000000.0 nu: 0.2 friction_angle: 25.0 cohesion: 0.0 requires_grad: false random: false name: sand ckpt: null shape: type: cube name: dataset center:
  • 0.5
  • 0.5
  • 0.5 size:
  • 0.5
  • 0.5
  • 0.5 resolution: 10 mode: uniform sort: null vel: random: false lin_vel:
  • 1.0
  • -1.5
  • -2.0 ang_vel:
  • 4.0
  • 4.0
  • 4.0 name: sand rho: 1000.0 span:
    • 0
    • 1000 clip_bound: 0.5 render: spp: 32 width: 512 height: 512 skip_frame: 25 bound: 1.75 mpm_mul: 6 sph_version: cuda_ad_rgb pcd_version: cuda_ad_rgb has_sphere_emitter: true fps: 10 sim: quality: low num_steps: 1000 gravity:
    • 0.0
    • -9.8
    • 0.0 bc: freeslip num_grids: 20 dt: 0.0005 bound: 3 eps: 1.0e-07 skip_frame: 1 train: teacher: strategy: cosine start_lambda: 25 end_lambda: 200 num_epochs: 300 batch_size: 128 elasticity_lr: 1.0 plasticity_lr: 0.1 elasticity_wd: 0.0 plasticity_wd: 0.0 elasticity_grad_max_norm: 0.1 plasticity_grad_max_norm: 0.1 name: sand/dataset seed: 0 cpu: 0 num_cpus: 128 gpu: 0 overwrite: false resume: false gt_name: null dataset: true

Warp 0.6.1 initialized: CUDA not available Devices: "cpu" | x86_64 Kernel cache: /home/xxxx/.cache/warp/0.6.1 Error executing job with overrides: ['overwrite=False', 'resume=False', 'gpu=0', 'cpu=0', 'env=sand', 'render=debug', 'sim=low', 'name=sand/dataset', 'dataset=True'] Traceback (most recent call last): File "/home/xxxx/xxxx_research/NCLaw/experiments/eval.py", line 137, in main() File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main _run_hydra( File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra _run_app( File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app run_and_report( File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report raise ex File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report return func() File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in lambda: hydra.run( File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/internal/hydra.py", line 132, in run = ret.return_value File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value raise self._return_value File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job ret.return_value = task_function(task_cfg) File "/home/xxxx/xxxx_research/NCLaw/experiments/eval.py", line 35, in main wp_device = wp.get_device(f'cuda:{cfg.gpu}') File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/warp/context.py", line 1676, in get_device return runtime.get_device(ident) File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/warp/context.py", line 1484, in get_device return self.device_map[ident] KeyError: 'cuda:0' env: blob: bsdf_pcd: type: diffuse reflectance: type: rgb value:

• 0.59607843
• 0.98431373
• 0.59607843 material: elasticity: cls: SigmaElasticity E: 300000.0 nu: 0.25 requires_grad: false random: false plasticity: cls: VonMisesPlasticity E: 300000.0 nu: 0.25 sigma_y: 5000.0 requires_grad: false random: false name: plasticine ckpt: null shape: type: cube name: dataset center:
  • 0.5
  • 0.5
  • 0.5 size:
  • 0.5
  • 0.5
  • 0.5 resolution: 10 mode: uniform sort: null vel: random: false lin_vel:
  • 1.0
  • -1.5
  • -2.0 ang_vel:
  • 4.0
  • 4.0
  • 4.0 name: plasticine rho: 1000.0 span:
    • 0
    • 1000 clip_bound: 0.5 render: spp: 32 width: 512 height: 512 skip_frame: 25 bound: 1.75 mpm_mul: 6 sph_version: cuda_ad_rgb pcd_version: cuda_ad_rgb has_sphere_emitter: true fps: 10 sim: quality: low num_steps: 1000 gravity:
    • 0.0
    • -9.8
    • 0.0 bc: freeslip num_grids: 20 dt: 0.0005 bound: 3 eps: 1.0e-07 skip_frame: 1 train: teacher: strategy: cosine start_lambda: 25 end_lambda: 200 num_epochs: 300 batch_size: 128 elasticity_lr: 1.0 plasticity_lr: 0.1 elasticity_wd: 0.0 plasticity_wd: 0.0 elasticity_grad_max_norm: 0.1 plasticity_grad_max_norm: 0.1 name: plasticine/dataset seed: 0 cpu: 0 num_cpus: 128 gpu: 0 overwrite: false resume: false gt_name: null dataset: true

Warp 0.6.1 initialized: CUDA not available Devices: "cpu" | x86_64 Kernel cache: /home/xxxx/.cache/warp/0.6.1 Error executing job with overrides: ['overwrite=False', 'resume=False', 'gpu=0', 'cpu=0', 'env=plasticine', 'render=debug', 'sim=low', 'name=plasticine/dataset', 'dataset=True'] Traceback (most recent call last): File "/home/xxxx/xxxx_research/NCLaw/experiments/eval.py", line 137, in main() File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main _run_hydra( File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra _run_app( File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app run_and_report( File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report raise ex File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report return func() File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in lambda: hydra.run( File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/internal/hydra.py", line 132, in run = ret.return_value File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value raise self._return_value File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job ret.return_value = task_function(task_cfg) File "/home/xxxx/xxxx_research/NCLaw/experiments/eval.py", line 35, in main wp_device = wp.get_device(f'cuda:{cfg.gpu}') File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/warp/context.py", line 1676, in get_device return runtime.get_device(ident) File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/warp/context.py", line 1484, in get_device return self.device_map[ident] KeyError: 'cuda:0' env: blob: bsdf_pcd: type: diffuse reflectance: type: rgb value:

• 0.45490196
• 0.8
• 0.95686275 material: elasticity: cls: VolumeElasticity E: 100000.0 nu: 0.3 mode: taichi requires_grad: false random: false plasticity: cls: SigmaPlasticity requires_grad: false random: false name: water ckpt: null shape: type: cube name: dataset center:
  • 0.5
  • 0.5
  • 0.5 size:
  • 0.5
  • 0.5
  • 0.5 resolution: 10 mode: uniform sort: null vel: random: false lin_vel:
  • 1.0
  • -1.5
  • -2.0 ang_vel:
  • 4.0
  • 4.0
  • 4.0 name: water rho: 1000.0 span:
    • 0
    • 1000 clip_bound: 0.5 render: spp: 32 width: 512 height: 512 skip_frame: 25 bound: 1.75 mpm_mul: 6 sph_version: cuda_ad_rgb pcd_version: cuda_ad_rgb has_sphere_emitter: true fps: 10 sim: quality: low num_steps: 1000 gravity:
    • 0.0
    • -9.8
    • 0.0 bc: freeslip num_grids: 20 dt: 0.0005 bound: 3 eps: 1.0e-07 skip_frame: 1 train: teacher: strategy: cosine start_lambda: 25 end_lambda: 200 num_epochs: 300 batch_size: 128 elasticity_lr: 1.0 plasticity_lr: 0.1 elasticity_wd: 0.0 plasticity_wd: 0.0 elasticity_grad_max_norm: 0.1 plasticity_grad_max_norm: 0.1 name: water/dataset seed: 0 cpu: 0 num_cpus: 128 gpu: 0 overwrite: false resume: false gt_name: null dataset: true

Warp 0.6.1 initialized: CUDA not available Devices: "cpu" | x86_64 Kernel cache: /home/xxxx/.cache/warp/0.6.1 Error executing job with overrides: ['overwrite=False', 'resume=False', 'gpu=0', 'cpu=0', 'env=water', 'render=debug', 'sim=low', 'name=water/dataset', 'dataset=True'] Traceback (most recent call last): File "/home/xxxx/xxxx_research/NCLaw/experiments/eval.py", line 137, in main() File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main _run_hydra( File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra _run_app( File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app run_and_report( File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report raise ex File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report return func() File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in lambda: hydra.run( File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/internal/hydra.py", line 132, in run = ret.return_value File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value raise self._return_value File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job ret.return_value = task_function(task_cfg) File "/home/xxxx/xxxx_research/NCLaw/experiments/eval.py", line 35, in main wp_device = wp.get_device(f'cuda:{cfg.gpu}') File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/warp/context.py", line 1676, in get_device return runtime.get_device(ident) File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/warp/context.py", line 1484, in get_device return self.device_map[ident] KeyError: 'cuda:0'` Btw, I'm sure the CUDA has been installed correctly follow your provided instruction, so could your give me some advices to tackle this issue? Maybe the warp is the trouble maker? Looking forward to your reply!!!

[XiaogeTian]

I got the same problem. I am not sure if that's the warp context problem. It cannot recognize the cuda path.

[amannier]

same problem

[amannier]

When I upgraded the warp version to 0.15.1, this issue was resolved. At the same time, replace the tape. py file with the one mentioned by the author in another issue.

[PingchuanMa]

Sorry for the late reply. I tried but I cannot reproduce the error on my end though. Could you attach more details about your workstation? Does cuda:0 works for your torch.device or it's only broken in warp?

[leung1010]

Thanks for your reply. system: Ubuntu 20.04.4 LTS GPU: NVIDIA-SMI 470.57.02 , 0 Quadro RTX 6000 In the "conda list": nvidia-cublas-cu11 11.10.3.66 pypi_0 pypi nvidia-cuda-cupti-cu11 11.7.101 pypi_0 pypi nvidia-cuda-nvrtc-cu11 11.7.99 pypi_0 pypi nvidia-cuda-runtime-cu11 11.7.99 pypi_0 pypi nvidia-cudnn-cu11 8.5.0.96 pypi_0 pypi nvidia-cufft-cu11 10.9.0.58 pypi_0 pypi nvidia-curand-cu11 10.2.10.91 pypi_0 pypi nvidia-cusolver-cu11 11.4.0.1 pypi_0 pypi nvidia-cusparse-cu11 11.7.4.91 pypi_0 pypi nvidia-nccl-cu11 2.14.3 pypi_0 pypi nvidia-nvtx-cu11 11.7.91 pypi_0 pypi python 3.10.14 torch 2.0.1 pypi_0 pypi torchaudio 2.0.2 pypi_0 pypi torchvision 0.15.2 pypi_0 pypi

Cuda:0 works well for torch Python 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.

import torch if torch.cuda.is_available(): ... print("CUDA is available") ... else: ... print("CUDA is not available") ... CUDA is available device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") print("Using device:", device) Using device: cuda:0

Actually, I also tried another new DELL workstation(nvidia driver:535, A6000), it reported the same error info.

---

From: Pingchuan Ma @.> Sent: Wednesday, August 28, 2024 01:55 To: PingchuanMa/NCLaw @.> Cc: @. @.>; Author @.***> Subject: Re: [PingchuanMa/NCLaw] Report a bug: warp?(codes can be run well in my workstation) (Issue #2)

This is an external email.

Sorry for the late reply. I tried but I cannot reproduce the error on my end though. Could you attach more details about your workstation? Does cuda:0 works for your torch.device or it's only broken in warp?

— Reply to this email directly, view it on GitHubhttps://github.com/PingchuanMa/NCLaw/issues/2#issuecomment-2313185786, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALYH3ZBMCO36AWKYZIYMNYDZTS4R5AVCNFSM6AAAAABJGNCWICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJTGE4DKNZYGY. You are receiving this because you authored the thread.Message ID: @.***>

[PingchuanMa]

did you install warp from scratch or from pypi? could you also attach the output of bash command which nvcc please?

[leung1010]

Yes. Nvcc version is 11.3. But another workstation is 11.8, both report same error.

On 30 Aug 2024, at 00:02, Pingchuan Ma @.***> wrote:



This is an external email.

did you install warp from scratch or from pypi? could you also attach the output of bash command which nvcc please?

— Reply to this email directly, view it on GitHubhttps://github.com/PingchuanMa/NCLaw/issues/2#issuecomment-2318235083, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALYH3ZFDCS42I2Q67I56DQTZT5AYHAVCNFSM6AAAAABJGNCWICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJYGIZTKMBYGM. You are receiving this because you authored the thread.Message ID: @.***>

[PingchuanMa]

I do presume that this is a warp installation problem. Could you try to compile warp from scratch? The warp version used in this repo is attached in third_party/warp. I will also try to reproduce this on a computer with cuda 11.x to see if it's a cuda problem. Will let you know.

[leung1010]

OK, thanks. l will let you know after once try.

Sent from my iPhone

On 31 Aug 2024, at 07:26, Pingchuan Ma @.***> wrote:



This is an external email.

I do presume that this is a warp installation problem. Could you try to compile warp from scratch? The warp version used in this repo is attached in third_party/warp. I will also try to reproduce this on a computer with cuda 11.x to see if it's a cuda problem. Will let you know.

— Reply to this email directly, view it on GitHubhttps://github.com/PingchuanMa/NCLaw/issues/2#issuecomment-2322595692, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALYH3ZEVEF3M3U5YSYSJO4TZUD5RPAVCNFSM6AAAAABJGNCWICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRSGU4TKNRZGI. You are receiving this because you authored the thread.Message ID: @.***>

[leung1010]

I do presume that this is a warp installation problem. Could you try to compile warp from scratch? The warp version used in this repo is attached in third_party/warp. I will also try to reproduce this on a computer with cuda 11.x to see if it's a cuda problem. Will let you know.

Hi pingchuan,

Sry for late reply. I didn't work these days. Right now, I ran pip uninstall -y warp warp-lang pip install warp-lang=0.15.1 It works well now.

Many thanks!