Closed chschroeder closed 4 months ago
Hi, congratulations to raising the first issue, I'm truly confused 🥲. We checked everything before uploading the code and I can't seem to reproduce the issue. I just installed everything from scratch, on my machine and everything works.
Only thing I spotted: you said you ran it on the webcage task, but in the logs it says '+task=db_aspect'
. If you somehow run multiple consecutive runs without reinitialization, I can see how this error occurs. I just pushed a fix that likely will help here, although I'm still not sure how it happened in the first place.
EDIT: we just noticed that the paths do indeed indicate webcage, but the logged CLI args do not. Do you by chance still have the commandline args you started the run with?
No worries, no repository is perfect. Issues being raised just means that your repo is being used ;).
And sorry, I didn't express the setup clearly. I ran one of the examples:
python src/train.py +train_args=a100 +model=gbert_base +task='glob(*)'
With this command, the program runs for a while and just the error occurs at exactly this task.
The error is raised due to model conf (L55) being undefined. This can only occur when peft_config.task_type != "???"
, which is defined a few lines above with exactly that default value. So you will not be able to reproduce it in isolation, this is a side effect of a previous run changing peft_config
.
This is not urgent for me. I just wanted to report it.
Ok, I see. I've removed this example from the readme for now. Running the tasks one by one should work fine. Until that's fixed, I would recommend going that route 😅
Thanks again for reporting this, as running multiple runs one after another using the glob
syntax is, I would say, the least-tested feature of this repo. I expected hydra to initialize completely separate interpreters for every run. As this is clearly not the case, I don't see how we can make reasonably sure that there are no unintended side effects between consecutive runs. Hence, I removed the example from the README and recommend against using it, as under this perspective, my "fix" only delays the issue until a different task type is sampled.
Hi,
thanks again for providing this awesome benchmark!
Installation worked fine, but my first test run with "gbert_base" resulted in an error on the webcage task. Looking at the position in the source code where the error is raised I can see that this can happen, however, I don't fully understand the logic here, so I cannot propose a fix. Still, I wanted to report the problem.
Logs: Here is an excerpt of the logs
``` <...> 2024-06-30 16:45:13.129 | INFO | __main__:inner_train:75 - data_base_dir: data/ seed: 42 debug: false hydra.job.chdir: true train_args: learning_rate: 5.0e-05 batch_size: 8 gradient_accumulation_steps: 1 epochs: 5 precision: full disable_qlora: false data_dir: /localdir/data/raw model: model_name: deepset/gbert-base task: task_name: webcage framework: flair multi_label: true embedding_type: TransformerDocumentEmbeddings classifier_type: TextTripleClassifier label_type: label classifier_args: label_type: label corpus_type: DataTripleCorpus corpus_args: data_folder: /home/cschroeder/workspaces/llm/SuperGLEBer/data/WebCAGe columns: - 0 - 1 - 2 - 3 label_type: label skip_first_line: false 2024-06-30T16:45:13.129732+0200 INFO data_base_dir: data/ seed: 42 debug: false hydra.job.chdir: true train_args: learning_rate: 5.0e-05 batch_size: 8 gradient_accumulation_steps: 1 epochs: 5 precision: full disable_qlora: false data_dir: /localdir/data/raw model: model_name: deepset/gbert-base task: task_name: webcage framework: flair multi_label: true embedding_type: TransformerDocumentEmbeddings classifier_type: TextTripleClassifier label_type: label classifier_args: label_type: label corpus_type: DataTripleCorpus corpus_args: data_folder: /home/cschroeder/workspaces/llm/SuperGLEBer/data/WebCAGe columns: - 0 - 1 - 2 - 3 label_type: label skip_first_line: false 2024-06-30 16:45:13.131 | INFO | __main__:inner_train:84 - Training with flair 2024-06-30T16:45:13.131294+0200 INFO Training with flair 2024-06-30 16:45:13.136 | INFO | train_flair:training:40 - cuda:0 2024-06-30T16:45:13.136567+0200 INFO cuda:0 2024-06-30 16:45:13.137 | INFO | train_flair:training:42 - loading corpus 2024-06-30T16:45:13.137756+0200 INFO loading corpus 2024-06-30 16:45:13,139 Reading data from /home/cschroeder/workspaces/llm/SuperGLEBer/data/WebCAGe 2024-06-30 16:45:13,139 Train: /home/cschroeder/workspaces/llm/SuperGLEBer/data/WebCAGe/train.txt 2024-06-30 16:45:13,139 Dev: /home/cschroeder/workspaces/llm/SuperGLEBer/data/WebCAGe/dev.txt 2024-06-30 16:45:13,139 Test: /home/cschroeder/workspaces/llm/SuperGLEBer/data/WebCAGe/test.txt 2024-06-30 16:45:15.846 | INFO | train_flair:training:47 - first sample: DataPair: 'Sentence[26]: "Die Null-Promille-Grenze gilt vor allem in osteuropäischen Ländern: Estland, Rumänien, Ungarn, Tschechien und die Slowakei erlauben gar keinen Alkohol am Steuer."' + 'Sentence[21]: "Drei Finger für mehr Höflichkeit: Mit diesem Handzeichen wollte der ADAC im Jahr 1968 die Wut am Steuer dämpfen."' + 'Sentence[1]: "Steuer"' → True (1.0) 2024-06-30T16:45:15.846091+0200 INFO first sample: DataPair: 'Sentence[26]: "Die Null-Promille-Grenze gilt vor allem in osteuropäischen Ländern: Estland, Rumänien, Ungarn, Tschechien und die Slowakei erlauben gar keinen Alkohol am Steuer."' + 'Sentence[21]: "Drei Finger für mehr Höflichkeit: Mit diesem Handzeichen wollte der ADAC im Jahr 1968 die Wut am Steuer dämpfen."' + 'Sentence[1]: "Steuer"' → True (1.0) 2024-06-30 16:45:15,847 Filtering empty sentences 2024-06-30 16:45:15,938 Corpus: 6249 train + 1032 dev + 2069 test sentences 2024-06-30 16:45:15,938 Computing label dictionary. Progress: 6249it [00:00, 80785.49it/s] 2024-06-30 16:45:16,016 Dictionary created for label 'label' with 3 values: False (seen 4775 times), True (seen 1474 times) 2024-06-30 16:45:16.024 | INFO | train_flair:training:69 - label distribution: defaultdict(