search

TorchMoE / MoE-Infinity

PyTorch library for cost-effective, fast and easy serving of MoE models.
Apache License 2.0
88 stars 5 forks source link

run on the mutiple gpus #15

Open YLSnowy opened 4 months ago

YLSnowy commented 4 months ago

Is there an unquantized version that can run on multiple GPUs?

drunkcoding commented 4 months ago

Feel free to replace the example model with "mistralai/Mixtral-8x7B-Instruct-v0.1". If there's further error in running multi-GPU version, please feel free to post here

Tingberer commented 1 month ago

when I run interface_example.py, I meet the errors below: Traceback (most recent call last): File "/home/admin/Documents/MoE-Infinity-main/examples/interface_example.py", line 32, in names = datasets.get_dataset_config_names(dataset_name) File "/home/admin/anaconda3/envs/moe-infinity/lib/python3.9/site-packages/datasets/inspect.py", line 347, in get_dataset_config_names dataset_module = dataset_module_factory( File "/home/admin/anaconda3/envs/moe-infinity/lib/python3.9/site-packages/datasets/load.py", line 1915, in dataset_module_factory raise e1 from None File "/home/admin/anaconda3/envs/moe-infinity/lib/python3.9/site-packages/datasets/load.py", line 1867, in dataset_module_factory can_load_config_from_parquet_export = "DEFAULT_CONFIG_NAME" not in f.read() File "/home/admin/anaconda3/envs/moe-infinity/lib/python3.9/codecs.py", line 322, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

drunkcoding commented 1 week ago

when I run interface_example.py, I meet the errors below: Traceback (most recent call last): File "/home/admin/Documents/MoE-Infinity-main/examples/interface_example.py", line 32, in names = datasets.get_dataset_config_names(dataset_name) File "/home/admin/anaconda3/envs/moe-infinity/lib/python3.9/site-packages/datasets/inspect.py", line 347, in get_dataset_config_names dataset_module = dataset_module_factory( File "/home/admin/anaconda3/envs/moe-infinity/lib/python3.9/site-packages/datasets/load.py", line 1915, in dataset_module_factory raise e1 from None File "/home/admin/anaconda3/envs/moe-infinity/lib/python3.9/site-packages/datasets/load.py", line 1867, in dataset_module_factory can_load_config_from_parquet_export = "DEFAULT_CONFIG_NAME" not in f.read() File "/home/admin/anaconda3/envs/moe-infinity/lib/python3.9/codecs.py", line 322, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

the dataset loading error is not related to any part of MoE-Infinity, would suggest delete local cache and retry