Empty Loss Tensor in G-Retriever Code Example

[giuseppefutia]

🐛 Describe the bug

Hello,

I tried to test the example related to the new G-Retriever model in colab: https://github.com/pyg-team/pytorch_geometric/blob/master/examples/llm/g_retriever.py.

To monitor the quality of the training, I checked the computed loss values for the training and the validation datasets. After line 139 (loss = get_loss(model, batch, model_save_name)), I added the following code block:

print(f"Loss: {loss.item()}")
print(loss)
print(type(loss))
print(loss.shape)
print(loss.dtype)

I achieved the following result:

Loss: nan
tensor(nan, device='cuda:0', grad_fn=<NllLossBackward0>)
<class 'torch.Tensor'>
torch.Size([])
torch.float32

I suspect there is an issue in the forward method of the GRetriever class: https://pytorch-geometric.readthedocs.io/en/stable/_modules/torch_geometric/nn/models/g_retriever.html#GRetriever.forward.

Thank you for your attention.

Versions

Collecting environment information... PyTorch version: 2.5.0+cu121 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: 14.0.0-1ubuntu1.1 CMake version: version 3.30.5 Libc version: glibc-2.35

Python version: 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-6.1.85+-x86_64-with-glibc2.35 Is CUDA available: True CUDA runtime version: 12.2.140 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: NVIDIA A100-SXM4-40GB Nvidia driver version: 535.104.05 cuDNN version: Probably one of the following: /usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.6 /usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.6 /usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.6 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.6 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.6 /usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.6 /usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.6 HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 12 On-line CPU(s) list: 0-11 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) CPU @ 2.20GHz CPU family: 6 Model: 85 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 1 Stepping: 7 BogoMIPS: 4400.42 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear arch_capabilities Hypervisor vendor: KVM Virtualization type: full L1d cache: 192 KiB (6 instances) L1i cache: 192 KiB (6 instances) L2 cache: 6 MiB (6 instances) L3 cache: 38.5 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-11 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Vulnerable Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers Vulnerability Spectre v2: Vulnerable; IBPB: disabled; STIBP: disabled; PBRSB-eIBRS: Vulnerable; BHI: Vulnerable (Syscall hardening enabled) Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Vulnerable

Versions of relevant libraries: [pip3] numpy==1.26.4 [pip3] nvidia-cublas-cu12==12.6.3.3 [pip3] nvidia-cuda-cupti-cu12==12.6.80 [pip3] nvidia-cuda-runtime-cu12==12.6.77 [pip3] nvidia-cudnn-cu12==9.5.1.17 [pip3] nvidia-cufft-cu12==11.3.0.4 [pip3] nvidia-curand-cu12==10.3.7.77 [pip3] nvidia-cusolver-cu12==11.7.1.2 [pip3] nvidia-cusparse-cu12==12.5.4.2 [pip3] nvidia-nccl-cu12==2.23.4 [pip3] nvidia-nvjitlink-cu12==12.6.77 [pip3] nvtx==0.2.10 [pip3] optree==0.13.0 [pip3] pynvjitlink-cu12==0.4.0 [pip3] torch==2.5.0+cu121 [pip3] torch_cluster==1.6.3+pt25cu121 [pip3] torch-geometric==2.6.1 [pip3] torch_scatter==2.1.2+pt25cu121 [pip3] torch_sparse==0.6.18+pt25cu121 [pip3] torch_spline_conv==1.2.2+pt25cu121 [pip3] torchaudio==2.5.0+cu121 [pip3] torchsummary==1.5.1 [pip3] torchvision==0.20.0+cu121 [conda] Could not collect

[Kh4L]

whats your command line?

[giuseppefutia]

I am running it in colab, so I needed to update the script to launch the run.

if __name__ == '__main__':
    args = SimpleNamespace(
      gnn_hidden_channels=1024,
      num_gnn_layers=4,
      lr=1e-5,
      epochs=2,
      batch_size=8,
      eval_batch_size=16,
      checkpointing=True,  # Set this to True or False as needed
      tiny_llama=True    # Set this to True or False as needed
  )

    start_time = time.time()
    train(
        args.epochs,
        args.gnn_hidden_channels,
        args.num_gnn_layers,
        args.batch_size,
        args.eval_batch_size,
        args.lr,
        checkpointing=args.checkpointing,
        tiny_llama=args.tiny_llama,
    )
    print(f"Total Time: {time.time() - start_time:2f}s")

Not sure if this is what you expect when you say "command line".