Closed anesaibr closed 2 months ago
Hi,
Thanks for your attention to our work! Using Legommenders to tune with Llama on a new dataset is a complex work. I hope I can help you make it.
The entire pipeline is as follows:
For now, I think the error occurs on the first step: tokenization.
I guess you followed the scripts in the process folder. It seems you first tokenze news data and save news IDs into nid
vocabulary. Next, when you tokenize interaction/user-sequence data, you also provide this nid
vocabulary. However, I guess there is unrecoginized news in your interaction/user-sequence data, so the nid
vocab is then extended to compat this data. Therefore, the later nid
vocab is larger than the original one.
In this case, I suggest you first filtering out all the invalid nid
s.
Feel free to ask any questions, thanks.
Thank you for your quick reply! Me and my team managed to fix the tokenization error and adjust the nid
vocab to fit with our data. However, we have now progressed to a new error which comes from the configuration.embed file since we are not sure how to generate the npy files which are present in the path
variable :
Hi, if you are using llama, please extract llama token embeddings first.
You can refer to the following scripts:
import numpy as np
from transformers import LlamaModel
pretrained_dir = '/path/to/llama/'
device = 'cuda:1'
# Load pre-trained model (weights)
model = LlamaModel.from_pretrained(pretrained_dir).to(device) # type: LlamaModel
print(len(model.layers))
# Put the model in "evaluation" mode, meaning feed-forward operation.
model.eval()
embeds = model.embed_tokens.weight.cpu().detach().numpy()
np.save('llama-token.npy', embeds)
BTW, if you meet OOM problem, please add --page_size 64
in the command line.
We are actually experiencing another issue when wanting to tune LLama(for training preparation) since our dataset does not contain a test folder (only train & dev). More specifically, we are not able to use test_llm_layer_split
inside the llama-split.yaml file and assumed we could use the other splits invloving train and dev sets. However, this currently gives an error which complains about a missing learning rate I suppose. Any suggestion or solution to this current error?
File "/gpfs/home3/scur1569/Legommenders/worker.py", line 489, in <module>
worker.run()
File "/gpfs/home3/scur1569/Legommenders/worker.py", line 441, in run
epoch = self.train_runner()
^^^^^^^^^^^^^^^^^^^
File "/gpfs/home3/scur1569/Legommenders/worker.py", line 366, in train_runner
pnt('use single lr:', self.exp.policy.lr)
File "/home/scur1569/.conda/envs/dire_tokenize/lib/python3.11/site-packages/pigmento/pigmento.py", line 83, in __call__
return self._call(*args, _caller_name=caller_name, _caller_class=caller_class, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/scur1569/.conda/envs/dire_tokenize/lib/python3.11/site-packages/pigmento/pigmento.py", line 120, in _call
text = ' '.join([str(arg) for arg in args])
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/scur1569/.conda/envs/dire_tokenize/lib/python3.11/site-packages/pigmento/pigmento.py", line 120, in <listcomp>
text = ' '.join([str(arg) for arg in args])
^^^^^^^^
File "/home/scur1569/.conda/envs/dire_tokenize/lib/python3.11/site-packages/oba/oba.py", line 34, in __str__
raise ValueError(f'Path {NoneObj.raw(self)} not exists')
ValueError: Path lr not exists```
Hi, please use test_llm_layer_split
. In the meantime, in your data/eb-nerd.yaml
, please set test depot path as same as dev depot path.
Are you participating in the RecSys challenge? If you need instant assistance these days, we can add each other on WeChat, Telegram, or WhatsApp.
We are participating in the Recsys challenge indeed! It would be great if we would keep in contact via WhatsApp! Can you perhaps share your number and I would then add you to our WhatsApp group?
Hi,
I have been refering to the ONCE project to test and run it on a news dataset ,namely, Ekstra Bladet News Recommendation Dataset. However, me and my team seem to be getting the error attached below when trying to run the Training Preparation (for tuning LLaMA which makes use of the configuration files of this repository. The below ouput is the generated configuration dict which is printed at the beginning of the output :
Moreover this is the following error that gets printed when running the [Main file (worker.py)]: (https://github.com/Jyonn/Legommenders/blob/4e8878e8188dc1fc880e9db055a800b5736d19df/worker.py) File "/gpfs/home3/scur1569/Legommenders/worker.py", line 488, in
worker = Worker(config=configuration)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/home3/scur1569/Legommenders/worker.py", line 58, in init
self.controller = Controller(
^^^^^^^^^^^
File "/gpfs/home3/scur1569/Legommenders/loader/controller.py", line 47, in init
self.depots = Depots(user_data=self.data.user, modes=self.modes, column_map=self.column_map)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/home3/scur1569/Legommenders/loader/depots.py", line 37, in init
depot.union(*[DepotHub.get(d) for d in user_data.union])
File "/home/scur1569/.conda/envs/dire_tokenize/lib/python3.11/site-packages/UniTok/unidep.py", line 221, in union
self.vocs = self._merge_vocs(self.vocs, depot.vocs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/scur1569/.conda/envs/dire_tokenize/lib/python3.11/site-packages/UniTok/unidep.py", line 189, in _merge_vocs
raise ValueError(f'vocab {name} config conflict')
ValueError: vocab nid config conflict