webcage: UnboundLocalError: local variable 'model_conf' referenced before assignment

[chschroeder]

Hi,

thanks again for providing this awesome benchmark!

Installation worked fine, but my first test run with "gbert_base" resulted in an error on the webcage task. Looking at the position in the source code where the error is raised I can see that this can happen, however, I don't fully understand the logic here, so I cannot propose a fix. Still, I wanted to report the problem.

Logs: Here is an excerpt of the logs

``` <...> 2024-06-30 16:45:13.129 | INFO | __main__:inner_train:75 - data_base_dir: data/ seed: 42 debug: false hydra.job.chdir: true train_args: learning_rate: 5.0e-05 batch_size: 8 gradient_accumulation_steps: 1 epochs: 5 precision: full disable_qlora: false data_dir: /localdir/data/raw model: model_name: deepset/gbert-base task: task_name: webcage framework: flair multi_label: true embedding_type: TransformerDocumentEmbeddings classifier_type: TextTripleClassifier label_type: label classifier_args: label_type: label corpus_type: DataTripleCorpus corpus_args: data_folder: /home/cschroeder/workspaces/llm/SuperGLEBer/data/WebCAGe columns: - 0 - 1 - 2 - 3 label_type: label skip_first_line: false 2024-06-30T16:45:13.129732+0200 INFO data_base_dir: data/ seed: 42 debug: false hydra.job.chdir: true train_args: learning_rate: 5.0e-05 batch_size: 8 gradient_accumulation_steps: 1 epochs: 5 precision: full disable_qlora: false data_dir: /localdir/data/raw model: model_name: deepset/gbert-base task: task_name: webcage framework: flair multi_label: true embedding_type: TransformerDocumentEmbeddings classifier_type: TextTripleClassifier label_type: label classifier_args: label_type: label corpus_type: DataTripleCorpus corpus_args: data_folder: /home/cschroeder/workspaces/llm/SuperGLEBer/data/WebCAGe columns: - 0 - 1 - 2 - 3 label_type: label skip_first_line: false 2024-06-30 16:45:13.131 | INFO | __main__:inner_train:84 - Training with flair 2024-06-30T16:45:13.131294+0200 INFO Training with flair 2024-06-30 16:45:13.136 | INFO | train_flair:training:40 - cuda:0 2024-06-30T16:45:13.136567+0200 INFO cuda:0 2024-06-30 16:45:13.137 | INFO | train_flair:training:42 - loading corpus 2024-06-30T16:45:13.137756+0200 INFO loading corpus 2024-06-30 16:45:13,139 Reading data from /home/cschroeder/workspaces/llm/SuperGLEBer/data/WebCAGe 2024-06-30 16:45:13,139 Train: /home/cschroeder/workspaces/llm/SuperGLEBer/data/WebCAGe/train.txt 2024-06-30 16:45:13,139 Dev: /home/cschroeder/workspaces/llm/SuperGLEBer/data/WebCAGe/dev.txt 2024-06-30 16:45:13,139 Test: /home/cschroeder/workspaces/llm/SuperGLEBer/data/WebCAGe/test.txt 2024-06-30 16:45:15.846 | INFO | train_flair:training:47 - first sample: DataPair: 'Sentence[26]: "Die Null-Promille-Grenze gilt vor allem in osteuropäischen Ländern: Estland, Rumänien, Ungarn, Tschechien und die Slowakei erlauben gar keinen Alkohol am Steuer."' + 'Sentence[21]: "Drei Finger für mehr Höflichkeit: Mit diesem Handzeichen wollte der ADAC im Jahr 1968 die Wut am Steuer dämpfen."' + 'Sentence[1]: "Steuer"' → True (1.0) 2024-06-30T16:45:15.846091+0200 INFO first sample: DataPair: 'Sentence[26]: "Die Null-Promille-Grenze gilt vor allem in osteuropäischen Ländern: Estland, Rumänien, Ungarn, Tschechien und die Slowakei erlauben gar keinen Alkohol am Steuer."' + 'Sentence[21]: "Drei Finger für mehr Höflichkeit: Mit diesem Handzeichen wollte der ADAC im Jahr 1968 die Wut am Steuer dämpfen."' + 'Sentence[1]: "Steuer"' → True (1.0) 2024-06-30 16:45:15,847 Filtering empty sentences 2024-06-30 16:45:15,938 Corpus: 6249 train + 1032 dev + 2069 test sentences 2024-06-30 16:45:15,938 Computing label dictionary. Progress: 6249it [00:00, 80785.49it/s] 2024-06-30 16:45:16,016 Dictionary created for label 'label' with 3 values: False (seen 4775 times), True (seen 1474 times) 2024-06-30 16:45:16.024 | INFO | train_flair:training:69 - label distribution: defaultdict(. at 0x7f1ba9d49d30>, {'True': 1474, 'False': 4775}) 2024-06-30T16:45:16.024165+0200 INFO label distribution: defaultdict(. at 0x7f1ba9d49d30>, {'True': 1474, 'False': 4775}) 2024-06-30 16:45:16.025 | INFO | train_flair:training:71 - creating model 2024-06-30T16:45:16.025466+0200 INFO creating model 2024-06-30 16:45:16.103 | INFO | train_flair:training:73 - weight_dict has been created {'False': 0.6543455497382199, 'True': 2.119742198100407} 2024-06-30T16:45:16.103006+0200 INFO weight_dict has been created {'False': 0.6543455497382199, 'True': 2.119742198100407} 2024-06-30 16:45:16.227 | INFO | utils:get_max_seq_length:21 - Using 512 as max_seq_length 2024-06-30T16:45:16.227806+0200 INFO Using 512 as max_seq_length Error executing job with overrides: ['+train_args=a100', '+model=gbert_base', '+task=db_aspect'] Traceback (most recent call last): File "/home/cschroeder/.pyenv/versions/3.9.18/lib/python3.9/unittest/mock.py", line 1336, in patched return func(*newargs, **newkeywargs) File "/home/cschroeder/workspaces/llm/SuperGLEBer/src/train.py", line 95, in training patched_train(cfg) File "/home/cschroeder/workspaces/llm/SuperGLEBer/src/train.py", line 40, in patched_train automodel_patched_train(cfg) File "/home/cschroeder/.pyenv/versions/3.9.18/lib/python3.9/unittest/mock.py", line 1336, in patched return func(*newargs, **newkeywargs) File "/home/cschroeder/workspaces/llm/SuperGLEBer/src/train.py", line 64, in automodel_patched_train inner_train(cfg) File "/home/cschroeder/workspaces/llm/SuperGLEBer/src/train.py", line 85, in inner_train train_flair(cfg) File "/home/cschroeder/workspaces/llm/SuperGLEBer/src/train_flair.py", line 89, in training embeddings=globals()[cfg["task"]["embedding_type"]]( File "/home/cschroeder/.pyenv/versions/supergleber/lib/python3.9/site-packages/flair/embeddings/document.py", line 47, in __init__ TransformerEmbeddings.__init__( File "/home/cschroeder/.pyenv/versions/supergleber/lib/python3.9/site-packages/flair/embeddings/transformer.py", line 991, in __init__ transformer_model = AutoModel.from_pretrained(model, config=config) File "/home/cschroeder/workspaces/llm/SuperGLEBer/src/utils.py", line 81, in new_from_pretrained model = get_LORA_model(model, cfg=cfg) File "/home/cschroeder/workspaces/llm/SuperGLEBer/src/utils.py", line 55, in get_LORA_model if "mistral" in model_conf.architectures[0].lower() or "mbart" in model_conf.architectures[0].lower(): UnboundLocalError: local variable 'model_conf' referenced before assignment Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace. ```

[janpf]

Hi, congratulations to raising the first issue, I'm truly confused 🥲. We checked everything before uploading the code and I can't seem to reproduce the issue. I just installed everything from scratch, on my machine and everything works.

Only thing I spotted: you said you ran it on the webcage task, but in the logs it says '+task=db_aspect'. If you somehow run multiple consecutive runs without reinitialization, I can see how this error occurs. I just pushed a fix that likely will help here, although I'm still not sure how it happened in the first place.

EDIT: we just noticed that the paths do indeed indicate webcage, but the logged CLI args do not. Do you by chance still have the commandline args you started the run with?

[chschroeder]

No worries, no repository is perfect. Issues being raised just means that your repo is being used ;).

And sorry, I didn't express the setup clearly. I ran one of the examples: python src/train.py +train_args=a100 +model=gbert_base +task='glob(*)'

With this command, the program runs for a while and just the error occurs at exactly this task.

The error is raised due to model conf (L55) being undefined. This can only occur when peft_config.task_type != "???", which is defined a few lines above with exactly that default value. So you will not be able to reproduce it in isolation, this is a side effect of a previous run changing peft_config.

This is not urgent for me. I just wanted to report it.

[janpf]

Ok, I see. I've removed this example from the readme for now. Running the tasks one by one should work fine. Until that's fixed, I would recommend going that route 😅

[janpf]

Thanks again for reporting this, as running multiple runs one after another using the glob syntax is, I would say, the least-tested feature of this repo. I expected hydra to initialize completely separate interpreters for every run. As this is clearly not the case, I don't see how we can make reasonably sure that there are no unintended side effects between consecutive runs. Hence, I removed the example from the README and recommend against using it, as under this perspective, my "fix" only delays the issue until a different task type is sampled.