Python Versions

[VedaRePowered]

I am trying to run export.py, but I'm running into the following error:

│Traceback (most recent call last):                                                                                                                           │
│  File "/home/veda/Downloads/llama2.rs/export.py", line 134, in <module>                                                                                     │
│    load_and_export("", output_path)                                                                                                                         │
│  File "/home/veda/Downloads/llama2.rs/export.py", line 126, in load_and_export                                                                              │
│    export(model, output_path)                                                                                                                               │
│  File "/home/veda/Downloads/llama2.rs/export.py", line 57, in export                                                                                        │
│    hidden_dim = model.layers[0].mlp.up_proj.build()[0].shape[1]                                                                                             │
│  File "/home/veda/miniconda3/envs/llama2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1695, in __getattr__                                │
│    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")                                                                        │
│AttributeError: 'GeneralQuantLinear' object has no attribute 'build'                                                                                         │

I believe this is related to having the incorrect version of some library, but I can't determine the correct version (I'm not really much of a python person), my versions are:

Python 3.10.9
torch==2.1.0.dev20230812+rocm5.5 
transformers==4.31.0
auto-gptq==0.3.0

I would suggest either updating the readme to include the exact versions to use, or adding a requirements.txt.

[srush]

Oh shoot, sorry. It looks like your version has EXPAND=True. Set that to False in the code.

Explanation: EXPAND was a debug mode that converts from quant to full. I'll document that file.

I'll make that a command line argument.

[VedaRePowered]

Hi thanks for the quick reply, I double checked the version of export.py that I have cloned and it does have EXPAND = False, additionally the line the error is on gets run regardless of that variable (the comment indicates it's needed for writing the file header)

Sorry if I'm missing something, and thanks again for the help

[srush]

Can you confirm that https://github.com/srush/llama2.rs/pull/18 fixed your issue?

[VedaRePowered]

Yes #18 has resolved my issue exporting the model, thanks again for all your help!