search

AlpinDale / misc-scripts

Miscellaneous scripts for various stuff.
GNU General Public License v3.0
2 stars 3 forks source link

GPTQ quantization #3

Open DxxtchGamer opened 9 months ago

DxxtchGamer commented 9 months ago

Hey, I've noticed that whenever I try to use GPTQ quantized models, it doesn't work. I've noticed it sees it as AWQ and not GPTQ, I don't know why that is because it worked fine yesterday. I tried to use "TheBloke/Toppy-M-7B-GPTQ", but it gives me this before it shuts down the notebook. 

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/endpoints/kobold/api_server.py", line 334, in <module>
    engine = AsyncAphrodite.from_engine_args(engine_args)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/async_aphrodite.py", line 486, in from_engine_args
    engine = cls(parallel_config.worker_use_ray,
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/async_aphrodite.py", line 270, in __init__
    self.engine = self._init_engine(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/async_aphrodite.py", line 306, in _init_engine
    return engine_class(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/aphrodite_engine.py", line 108, in __init__
    self._init_workers(distributed_init_method)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/aphrodite_engine.py", line 144, in _init_workers
    self._run_workers(
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/aphrodite_engine.py", line 747, in _run_workers
    self._run_workers_in_batch(workers, method, *args, **kwargs))
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/aphrodite_engine.py", line 720, in _run_workers_in_batch
    output = executor(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/task_handler/worker.py", line 72, in load_model
    self.model_runner.load_model()
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/task_handler/model_runner.py", line 36, in load_model
    self.model = get_model(self.model_config)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/modeling/loader.py", line 76, in get_model
    quant_config = get_quant_config(model_config.quantization,
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/modeling/hf_downloader.py", line 94, in get_quant_config
    return quant_cls.from_config(hf_quant_config)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/modeling/layers/quantization/awq.py", line 67, in from_config
    zero_point = cls.get_from_keys(config, ["zero_point"])
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/modeling/layers/quantization/base_config.py", line 50, in get_from_keys
    raise ValueError(f"Cannot find any of {keys} in the model's "
ValueError: Cannot find any of ['zero_point'] in the model's quantization config
AlpinDale commented 9 months ago

Yeah my bad I forgot to change that part. You can edit the notebook in colab and change the quantization name from awq to gptq. I'll fix this later so we can have an option for selecting GPTQ/AWQ/SqueezeLLM.

DxxtchGamer commented 9 months ago

Yeah my bad I forgot to change that part. You can edit the notebook in colab and change the quantization name from awq to gptq. I'll fix this later so we can have an option for selecting GPTQ/AWQ/SqueezeLLM.

Yeah, I tried that, but the output isn't what it should be, it's also giving me a lot of warnings, and the output isn't what it used to be. But I can wait until later so you're done fixing it, I just thought I should let you know.

These are the repeating warnings I get (I'm not sure what they mean):

WARNING 12-13 18:04:10 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:11 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:11 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:11 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:11 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:11 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:11 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:11 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:12 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:12 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:12 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:12 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:12 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:12 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:12 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:12 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:13 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:13 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:13 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:13 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:13 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:13 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:13 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:13 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:14 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:14 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:14 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:14 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:14 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:14 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:14 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:14 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:15 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:16 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:16 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:16 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:16 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:16 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:16 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:16 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:16 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:16 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:17 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:17 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:17 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:17 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:17 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:17 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:17 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:18 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:18 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:18 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:18 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:18 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:18 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:18 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:18 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:18 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:19 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:19 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:19 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:19 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:19 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:19 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:19 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:19 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:19 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:20 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:20 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:20 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:20 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:20 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:20 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:20 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:21 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:21 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:21 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:21 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:21 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:21 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:21 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:21 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:21 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:22 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:22 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:22 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:22 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:22 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:22 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:22 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:22 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:23 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:23 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:23 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:23 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:23 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:23 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:23 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:24 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:24 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:24 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:24 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:24 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:24 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:24 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:24 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:24 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:25 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:25 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:25 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:25 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:25 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:25 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:25 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:26 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:26 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:26 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:26 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:26 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:26 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:26 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:26 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:27 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:27 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:27 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:27 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:27 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:27 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:27 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:28 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:28 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:28 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:28 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:28 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:28 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:28 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:28 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:28 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:29 linear.py:306] Loading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:29 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:29 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
WARNING 12-13 18:04:29 linear.py:429] Loading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.
AlpinDale commented 9 months ago

Yeah that's normal for now, it'll work fine. I'm updating the notebook now. I'll also add a toggle to switch between Kobold and OpenAI API.

AlpinDale commented 9 months ago

I've updated the notebook.