Code is not running correctly on my supercomputer:

[wangjiajiTHU]

Code is not running correctly on my supercomputer: /sqfs/work/G15408/v60646/conda_env/nclaw) [v60646@squidhpc3 train]$ python invariant_full_meta-invariant_full_meta.py env: blob: bsdf_pcd: type: diffuse reflectance: type: rgb value:

• 0.92941176
• 0.32941176
• 0.23137255 material: elasticity: cls: InvariantFullMetaElasticity layer_widths:
• 64
• 64 norm: null nonlinearity: gelu no_bias: true normalize_input: true requires_grad: true plasticity: cls: InvariantFullMetaPlasticity layer_widths:
• 64
• 64 norm: null alpha: 0.001 nonlinearity: gelu no_bias: true normalize_input: true requires_grad: true name: jelly ckpt: null shape: type: cube name: dataset center:
  • 0.5
  • 0.5
  • 0.5 size:
  • 0.5
  • 0.5
  • 0.5 resolution: 10 mode: uniform sort: null vel: random: false lin_vel:
  • 1.0
  • -1.5
  • -2.0 ang_vel:
  • 4.0
  • 4.0
  • 4.0 name: jelly rho: 1000.0 span:
    • 0
    • 1000 clip_bound: 0.5 render: spp: 32 width: 512 height: 512 skip_frame: 25 bound: 1.75 mpm_mul: 6 sph_version: cuda_ad_rgb pcd_version: cuda_ad_rgb has_sphere_emitter: true fps: 10 sim: quality: low num_steps: 1000 gravity:
    • 0.0
    • -9.8
    • 0.0 bc: freeslip num_grids: 20 dt: 0.0005 bound: 3 eps: 1.0e-07 skip_frame: 1 train: teacher: strategy: cosine start_lambda: 25 end_lambda: 200 num_epochs: 300 batch_size: 128 elasticity_lr: 1.0 plasticity_lr: 0.1 elasticity_wd: 0.0 plasticity_wd: 0.0 elasticity_grad_max_norm: 0.1 plasticity_grad_max_norm: 0.1 name: jelly/train/invariant_full_meta-invariant_full_meta seed: 0 cpu: 0 num_cpus: 128 gpu: 0 overwrite: false resume: false

Warp 0.11.0 initialized: CUDA Toolkit: 11.5, Driver: 12.0 Devices: "cpu" | x86_64 "cuda:0" | Quadro RTX 6000 (sm_75) Kernel cache: /sqfs/home/v60646/.cache/warp/0.11.0 target directory (/sqfs2/cmc/1/work/G15408/v60646/github/NCLaw/experiments/log/jelly/train/invariant_full_meta-invariant_full_meta) already exists, overwrite? [Y/r/n] y overwriting directory (/sqfs2/cmc/1/work/G15408/v60646/github/NCLaw/experiments/log/jelly/train/invariant_full_meta-invariant_full_meta) 0%| | 0/1000 [00:00<?, ?it/s]/sqfs/work/G15408/v60646/conda_env/nclaw/lib/python3.10/site-packages/warp/torch.py:159: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /opt/conda/conda-bld/pytorch_1704987280714/work/build/aten/src/ATen/core/TensorBody.h:489.) if t.grad is None: 0%| | 0/300 [00:04<?, ?it/s] Error executing job with overrides: ['overwrite=False', 'resume=False', 'gpu=0', 'cpu=0', 'env=jelly', 'env/blob/material/elasticity=invariant_full_meta', 'env/blob/material/plasticity=invariant_full_meta', 'env.blob.material.elasticity.requires_grad=True', 'env.blob.material.plasticity.requires_grad=True', 'render=debug', 'sim=low', 'name=jelly/train/invariant_full_meta-invariant_full_meta'] Traceback (most recent call last): File "/sqfs2/cmc/1/work/G15408/v60646/github/NCLaw/experiments/train.py", line 131, in main loss.backward() File "/sqfs/work/G15408/v60646/conda_env/nclaw/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward torch.autograd.backward( File "/sqfs/work/G15408/v60646/conda_env/nclaw/lib/python3.10/site-packages/torch/autograd/init.py", line 266, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass File "/sqfs/work/G15408/v60646/conda_env/nclaw/lib/python3.10/site-packages/torch/autograd/function.py", line 289, in apply return user_fn(self, *args) File "/sqfs2/cmc/1/work/G15408/v60646/github/NCLaw/nclaw/sim/interface.py", line 62, in backward model.backward(statics, state_curr, state_next, tape) File "/sqfs2/cmc/1/work/G15408/v60646/github/NCLaw/nclaw/sim/mpm.py", line 313, in backward tape.backward() File "/sqfs/work/G15408/v60646/conda_env/nclaw/lib/python3.10/site-packages/warp/tape.py", line 119, in backward adj_inputs.append(self.get_adjoint(a)) File "/sqfs2/cmc/1/work/G15408/v60646/github/NCLaw/nclaw/warp/tape.py", line 24, in get_adjoint adj = wp.codegen.StructInstance(a.struct) AttributeError: 'NewStructInstance' object has no attribute 'struct'

[PingchuanMa]

Sorry for the late reply! Please try to replace the tape.py file with this file. The problem roots in the incompatibility of warp when upgrading from 0.6.1 to what you were using (0.11.0) from what I saw. Let me know if it helps.