search

SawanKumar28 / nile

NILE : Natural Language Inference with Faithful Natural Language Explanations
Apache License 2.0
29 stars 5 forks source link

Finetune crashing at loss.backward() #1

Closed vibhavagarwal5 closed 4 years ago

vibhavagarwal5 commented 4 years ago

Hi,

Just trying to run the code but the finetune_lm.py is crashing at loss.backward()

Stack trace:

Epoch:   0%|                                                                                                                                          | 0/3 [00:00<?, ?it/sTraceback (most recent call last):                                                                                                                 | 0/87472 [00:00<?, ?it/s]
  File "finetune_lm.py", line 553, in <module>
    main()
  File "finetune_lm.py", line 507, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "finetune_lm.py", line 157, in train
    loss.backward()
  File "/home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/autograd/__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)` (createCublasHandle at /opt/conda/conda-bld/pytorch_1587428266983/work/aten/src/ATen/cuda/CublasHandlePool.cpp:8)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x4e (0x7f6cca012b5e in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xdba405 (0x7f6ccaff9405 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #2: at::cuda::getCurrentCUDABlasHandle() + 0x94c (0x7f6ccaffa1ec in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #3: <unknown function> + 0xdafb01 (0x7f6ccafeeb01 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x1263db7 (0x7f6ccb4a2db7 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #5: THCudaTensor_addmm + 0x5c (0x7f6ccb4a84ac in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xea5f28 (0x7f6ccb0e4f28 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdc92e8 (0x7f6ccb0082e8 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #8: <unknown function> + 0xe224d0 (0x7f6cf5c264d0 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #9: <unknown function> + 0x29f9d0e (0x7f6cf77fdd0e in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #10: <unknown function> + 0xe224d0 (0x7f6cf5c264d0 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #11: at::Tensor::mm(at::Tensor const&) const + 0xf0 (0x7f6cf57ea180 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #12: <unknown function> + 0x264517c (0x7f6cf744917c in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #13: torch::autograd::generated::MmBackward::apply(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) + 0x151 (0x7f6cf7449f81 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #14: <unknown function> + 0x2ae8215 (0x7f6cf78ec215 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #15: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&) + 0x16f3 (0x7f6cf78e9513 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #16: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&, bool) + 0x3d2 (0x7f6cf78ea2f2 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #17: torch::autograd::Engine::thread_init(int) + 0x39 (0x7f6cf78e2969 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #18: torch::autograd::python::PythonEngine::thread_init(int) + 0x38 (0x7f6cfac29558 in /home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #19: <unknown function> + 0xc819d (0x7f6d1200b19d in /mnt/c7cfa338-89cd-4d15-b0b9-f1befc9a2c68/vibhav/anaconda3/envs/vesnli/bin/../lib/libstdc++.so.6)
frame #20: <unknown function> + 0x76db (0x7f6d269a16db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #21: clone + 0x3f (0x7f6d266caa3f in /lib/x86_64-linux-gnu/libc.so.6)