quic-mangal
commented
11 months ago We do support them for PyTorch Framework. An example would be- https://docs.qualcomm.com/bundle/publicresource/topics/80-64748-1/model_updates.html
Open wminiboy opened 11 months ago
quic-mangal
commented
11 months ago We do support them for PyTorch Framework. An example would be- https://docs.qualcomm.com/bundle/publicresource/topics/80-64748-1/model_updates.html
wminiboy
commented
11 months ago @quic-mangal Thanks for your reply , the link example is a image process for qat . but not a "NLP model"（Natural Language Processing）model ，it is Different to process the parameters 。
quic-mangal
commented
11 months ago @wminiboy, stable diffusion model's do take in text and have a U-net based architecture. That being said, we do support quantization of NLP models. This was the closest example I could find, but you could try running a model using regular workflow, nothing special is needed from a user's side for NLP models.
wminiboy
commented
11 months ago @quic-mangal Thanks for you reply , but I encountered many problems when conducting QAT on the NLP model。
This is the dummy_input:
input_shape = (1,256) input_ids = torch.randint(low=0, high=1000, size=input_shape) attention_mask = torch.zeros(input_shape, dtype=torch.long) token_type_ids = torch.ones(input_shape, dtype=torch.long)
dummy_input = (input_ids, attention_mask, token_type_ids)
1, when I called QuantizationSimModel(...) ，the following error occurred:
Traceback (most recent call last):
File "examples/training_sup_text_matching_model_qat.py", line 161, in list, use a tuple instead. for dict, use a NamedTuple instead). If you absolutely need this and know the side effects, pass strict=False to trace() to allow this behavior.
To fix it , I pass strict=False to trace()
2，when I called apply_cross_layer_equalization(...) ,the following error occurred:
File "examples/training_sup_text_matching_model_qat.py", line 259, in
To fix it , My modifications are as follows：
def create_rand_tensors_given_shapes(input_shape: Union[Tuple, List[Tuple]], device: torch.device) \ -> List[torch.Tensor]: """ Given shapes of some tensors, create one or more random tensors and return them as a list of tensors :param input_shape: Shapes of tensors to create :param device: Device to create tensors on :return: Created list of tensors """ if isinstance(input_shape, List): input_shapes = input_shape else: input_shapes = [input_shape]
rand_tensors = []
for shape in input_shapes:
#Modified by wminboy
#rand_tensors.append(torch.rand(shape).to(device))
rand_tensors.append(torch.randint(low=0, high=1000, size=shape).to(device))
rand_tensors.append(torch.zeros(shape, dtype=torch.long).to(device))
rand_tensors.append(torch.ones(shape, dtype=torch.long).to(device))
#done
return rand_tensors3, when I called apply_bias_correction(...) , the following error occurred:
File "examples/training_sup_text_matching_model_qat.py", line 259, in
To fix it , My modifications are as follows：
def forward_pass(model: torch.nn.Module, batch: torch.Tensor): """ forward pass depending model allocation on CPU / GPU till StopForwardException :param model: model :param batch: batch :return: Nothing """
if utils.is_model_on_gpu(model):
batch = batch.cuda()
try:
with utils.in_eval_mode(model), torch.no_grad():
#Modified by wminboy
#_ = model(batch)
if isinstance(batch, (list, tuple)):
_ = model(*batch)
else:
_ = model(batch)
#done
except StopForwardException:
passThen the following error occurred:
Traceback (most recent call last):
File "examples/training_sup_text_matching_model_qat.py", line 259, in
But I don't know how to solve this issue。
So, is it because of the input parameters that caused these errors? How should parameters be set? Your advice would be recommended greatly. Thanks!
quic-kyunggeu
commented
11 months ago Hi @wminiboy
As for the first error, it seems like you are using a dict as dummy_input. Unfortunately, PyTorch JIT tracer, which QuantizationSimModel relies on, is not good at handling dict inputs. I suggest using tuple as dummy_input instead of dict.
As for the second and third errors, first I have to admit that we have not yet tested CLE or bias correction rigorously against LLMs. Your custom solution for the second error seems good enough, but for the third error, we don't have a good insight yet. We're actively working on supporting LLMs, and we expect to resolve these issues in the future releases.
wminiboy
commented
11 months ago @quic-kyunggeu Thanks for you reply The first error, I found the reason , it is not cased by dummy_input, is becase the model output is not a tuple, so before call QuantizationSimModel(......) , I set model.config.return_dict = False .
The issuse solved ！
As for the third error, we look forward to your update ！ Thanks !
I would like to confirm the latest version of aimet. is support perform QAT on NLP models, such as the bert model? If so , Can you provide an example ？
I encountered many problems when conducting QAT on the NLP model。