RuntimeError when running guide_gligen.py for inference

Hello,

I am trying to do inference using the commands given in the README. Specifically, I run the following:

python guide_gligen.py --ckpt gligen_checkpoints/diffusion_pytorch_model.bin --file_save myfolder/ --type counting --box_pickle data_evaluate_LLM/gpt_generated_box/counting.p

The code runs for sometime. In myfolder, it creates folders myfolder_box and myfolder_image. Each of these new folders contain 4 images (apples, baseball, coffee, parking meter). Then, the code terminates. I am attaching the error log along with few logs just before the error occurs:

step  49
optimize 49
[3]
box available 4
o_names ['mouse', 'kitchen_floor', 'air']
position [[4], [8], [9], [15]] a small brown mouse scurries across the kitchen floor, pausing briefly to sniff the air before continuing on its way.
step  0
optimize 0
min, max tensor(0.0122, device='cuda:0', grad_fn=<MaxBackward1>) tensor(0.7187, device='cuda:0', grad_fn=<MaxBackward1>)
loss tensor(29.8951, device='cuda:0', grad_fn=<AddBackward0>) tensor(1.9124, device='cuda:0', grad_fn=<MulBackward0>) tensor(25.7252, device='cuda:0', grad_fn=<MulBackward0>) tensor(2.2575, grad_fn=<MulBackward0>) tensor(0.6265, device='cuda:0', grad_fn=<DivBackward0>)
Traceback (most recent call last):
  File "guide_gligen.py", line 587, in <module>
    run(meta, models, info_files, args, starting_noise, k,i)
  File "guide_gligen.py", line 440, in run
    samples_fake = sampler.sample(S=steps, shape=shape, input=input,  uc=uc, guidance_scale=config.guidance_scale, mask=inpainting_mask, x0=z0, loss_type=args.loss_type)
  File "/data/saumgupta/diffusion-repositories/attention-refocusing/ldm/models/diffusion/plms.py", line 63, in sample
    return self.plms_sampling(shape, input, uc, guidance_scale, mask=mask, x0=x0, loss_type=loss_type)
  File "/data/saumgupta/diffusion-repositories/attention-refocusing/ldm/models/diffusion/plms.py", line 107, in plms_sampling
    x = self.update_loss_self_cross( input,i, index, ts )
  File "/data/saumgupta/diffusion-repositories/attention-refocusing/ldm/models/diffusion/plms.py", line 174, in update_loss_self_cross
    grad_cond = torch.autograd.grad(loss.requires_grad_(True), [x])[0]  
  File "/home/saumgupta/.conda/envs/ldm_layout/lib/python3.8/site-packages/torch/autograd/__init__.py", line 436, in grad
    result = _engine_run_backward(
  File "/home/saumgupta/.conda/envs/ldm_layout/lib/python3.8/site-packages/torch/autograd/graph.py", line 769, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: _Map_base::at

I am wondering if any of you have encountered the issue before, or have any suggestions on how I could proceed. Since the code ran for a few images, I was not expecting this to occur. Thank you so much for your time, and great work! :D

Attention-Refocusing / attention-refocusing

RuntimeError when running guide_gligen.py for inference #13