search

dilab-zju / self-speculative-decoding

Code associated with the paper **Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding**
Apache License 2.0
117 stars 8 forks source link

some problems occured when execuating file search.ipynb #14

Closed hunzhizi closed 4 months ago

hunzhizi commented 4 months ago

I used the llama from Llama-2-7b-ms model's weight to load the whole model. when execuate search.ipynb's this line layer_searching.search(1000) I met some error. How can I fix it？I am new to this field and I am eagerly looking forward to your response.

AttributeError                            Traceback (most recent call last)
Cell In[21], [line 1](vscode-notebook-cell:?execution_count=21&line=1)
----> [1](vscode-notebook-cell:?execution_count=21&line=1) layer_searching.search(1000)

File /mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/searching.py:85, in LayerSkippingSearching.search(self, n_iter)
     [84](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/searching.py:84) def search(self, n_iter=1000):
---> [85](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/searching.py:85)     self.optimizer.maximize(init_points=0, n_iter=n_iter)
     [86](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/searching.py:86)     return self.get_solution()

File [~/.conda/envs/transformers/lib/python3.9/site-packages/bayes_opt/bayesian_optimization.py:310](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/~/.conda/envs/transformers/lib/python3.9/site-packages/bayes_opt/bayesian_optimization.py:310), in BayesianOptimization.maximize(self, init_points, n_iter, acquisition_function, acq, kappa, kappa_decay, kappa_decay_delay, xi, **gp_params)
    [308](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/~/.conda/envs/transformers/lib/python3.9/site-packages/bayes_opt/bayesian_optimization.py:308)     x_probe = self.suggest(util)
    [309](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/~/.conda/envs/transformers/lib/python3.9/site-packages/bayes_opt/bayesian_optimization.py:309)     iteration += 1
--> [310](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/~/.conda/envs/transformers/lib/python3.9/site-packages/bayes_opt/bayesian_optimization.py:310) self.probe(x_probe, lazy=False)
    [312](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/~/.conda/envs/transformers/lib/python3.9/site-packages/bayes_opt/bayesian_optimization.py:312) if self._bounds_transformer and iteration > 0:
    [313](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/~/.conda/envs/transformers/lib/python3.9/site-packages/bayes_opt/bayesian_optimization.py:313)     # The bounds transformer should only modify the bounds after
    [314](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/~/.conda/envs/transformers/lib/python3.9/site-packages/bayes_opt/bayesian_optimization.py:314)     # the init_points points (only for the true iterations)
    [315](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/~/.conda/envs/transformers/lib/python3.9/site-packages/bayes_opt/bayesian_optimization.py:315)     self.set_bounds(
    [316](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/~/.conda/envs/transformers/lib/python3.9/site-packages/bayes_opt/bayesian_optimization.py:316)         self._bounds_transformer.transform(self._space))

File [~/.conda/envs/transformers/lib/python3.9/site-packages/bayes_opt/bayesian_optimization.py:208](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/~/.conda/envs/transformers/lib/python3.9/site-packages/bayes_opt/bayesian_optimization.py:208), in BayesianOptimization.probe(self, params, lazy)
    [206](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/~/.conda/envs/transformers/lib/python3.9/site-packages/bayes_opt/bayesian_optimization.py:206)     self._queue.add(params)
    [207](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/~/.conda/envs/transformers/lib/python3.9/site-packages/bayes_opt/bayesian_optimization.py:207) else:
--> [208](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/~/.conda/envs/transformers/lib/python3.9/site-packages/bayes_opt/bayesian_optimization.py:208)     self._space.probe(params)
    [209](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/~/.conda/envs/transformers/lib/python3.9/site-packages/bayes_opt/bayesian_optimization.py:209)     self.dispatch(Events.OPTIMIZATION_STEP)
...
   [1268](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/~/.conda/envs/transformers/lib/python3.9/site-packages/torch/nn/modules/module.py:1268)         return modules[name]
-> [1269](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/~/.conda/envs/transformers/lib/python3.9/site-packages/torch/nn/modules/module.py:1269) raise AttributeError("'{}' object has no attribute '{}'".format(
   [1270](https://vscode-remote+ssh-002dremote-002b3090ti.vscode-resource.vscode-cdn.net/mnt/data/zhouShaoRepo/transformers/self-speculative-decoding-main/~/.conda/envs/transformers/lib/python3.9/site-packages/torch/nn/modules/module.py:1270)     type(self).__name__, name))

AttributeError: 'LlamaModel' object has no attribute '_prepare_decoder_attention_mask'
Output is truncated. View as a [scrollable element](command:cellOutput.enableScrolling?85768e79-dee1-4bef-93bf-03258f3c5fc8) or open in a [text editor](command:workbench.action.openLargeOutput?85768e79-dee1-4bef-93bf-03258f3c5fc8). Adjust cell output [settings](command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D)...
junzhang-zj commented 4 months ago

@hunzhizi tend to think that this is a version problem of transformers. Please try installing transformers==4.33.1.