add device_map and quantization_config

[mit1280]

Great work!

I was trying to use TextClassifier with microsoft/Phi-3-mini-128k-instruct but couldn't load model in free colab.

I made very simple changes which will allow TextClassifier to load big model to free colab with device_map="auto" and quantization_config.

You can check my notebook which has my changes.

[mit1280]

Hi @MVYaroshenko,

First point, deleted.

Second point, I added device_map new argument because device and device_map can have different values. For example, when I want to use GPU and CPU then I need to set device_map to "auto" and device to "cuda".

Third point, added docstring.

[mit1280]

No problem, you are correct that device is only used by tokenizer apart from model initialization.

If we have only device and use it in device_map then it will not work with tokenizer. We need "auto" value for device_map to load model on GPU and CPU but tokenizer doesn't support it.

I hope this make sense to have new argument as device_map.

[MVYaroshenko]

Thank you for your response. I understand that. I mean that the device is a subset of device_map (i.e., device_map includes device values and specific values such as dict and "auto"). So, can we use the value of device instead of device_map if device_map is not provided (equals None)?

[MVYaroshenko]

Something like this: device_map = self.device_map or self.device

[mit1280]

thanks for your input. Set device as default device_map if device_map is None.

[MVYaroshenko]

Thank you very much.