search

zhou13 / lcnn

LCNN: End-to-End Wireframe Parsing
MIT License
506 stars 94 forks source link

Can we speed up with multi-gpus? #41

Closed HC-2016 closed 3 years ago

HC-2016 commented 3 years ago

I try to speed up with multi-gpus as the followings:

    checkpoint = torch.load(args["<checkpoint>"])
    model = MultitaskLearner(model)
    model = LineVectorizer(model)
    model.load_state_dict(checkpoint["model_state_dict"])
    #model = model.to(device)
    model = nn.DataParallel(model, device_ids=[0, 1])
    model = model.to(device)
    model.eval()

But it throw errors: /tmp/pip-req-build-ocx5vxk7/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)->auto::operator()(int)->auto: block: [0,0,0], thread: [93,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. /tmp/pip-req-build-ocx5vxk7/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)->auto::operator()(int)->auto: block: [0,0,0], thread: [94,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. /tmp/pip-req-build-ocx5vxk7/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)->auto::operator()(int)->auto: block: [0,0,0], thread: [95,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. Traceback (most recent call last): File "inference.py", line 212, in <module> main() File "inference.py", line 181, in main H = model(input_dict)["preds"] File "lcnn/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__ result = self.forward(*input, **kwargs) File "lcnn/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward outputs = self.parallel_apply(replicas, inputs, kwargs) File "lcnn/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)]) File "lcnn/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply output.reraise() File "lcnn/lib/python3.6/site-packages/torch/_utils.py", line 385, in reraise raise self.exc_type(msg) RuntimeError: Caught RuntimeError in replica 0 on device 0. Original Traceback (most recent call last): File "lcnn/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker output = module(*input, **kwargs) File "lcnn/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__ result = self.forward(*input, **kwargs) File "lcnn/models/line_vectorizer.py", line 59, in forward meta, h["jmap"][i], h["joff"][i], input_dict["mode"] File "/lcnn/models/line_vectorizer.py", line 243, in sample_lines u, v, label = u[c], v[c], label[c] RuntimeError: copy_if failed to synchronize: device-side assert triggered

zhou13 commented 3 years ago

Unfortunately, it is not that easy. The sampling module and the dataset also need to be modified carefully.