Closed LeeVinteuil closed 3 years ago
Thank you for your asking. Can you post your code here?
The correct format of calling QAT should be like the example below.
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.5)
quantizer = QAT_Quantizer(model, configure_list, optimizer)
quantizer.compress()
model.to(device)
If you map your model to cuda previous to using quantizer.compress()
, this error may occured. And when you train your model or do the inference, all of your data should be mapped to cuda first.
You can check whether your implementation has problems mentioned above. If not, you can describe your inplementation more specifically so that I can help you find the problem.
Thank you very much for both your contribution and answer. My implementation has problems as you mentioned. This problem has been solved.
closing the issue as the problem has been solved.
Hi~ @linbinskn , I meet the same problem during using DataParallel
, and I found QAT_Quantizer.compress()
is inherited from Compressor.compress()
which does nothing but just return self.bound_model
. So what is meaning by calling quantizer.compress()
before model.to('cuda')
? And how can I solve the problem of RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
, really thanks if you have any advice.
Maybe, when you are building model, you can set device_map like:
def build_model(pretrained_model_name_or_path: str, task_name: str):
is_regression = task_name == 'stsb'
num_labels = 1 if is_regression else (3 if task_name == 'mnli' else 2)
model = BertForSequenceClassification.from_pretrained(pretrained_model_name_or_path, num_labels=num_labels,device_map="cuda:1")
return model
input_variable and model have been mapped to cuda,while the RuntimeError still occured.
File "/lib/python3.6/site-packages/nni/compression/torch/quantization/quantizers.py", line 59, in update_ema biased_ema = biased_ema decay + (1 - decay) value RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
this happens in QAT Quantizer