Closed agarfau closed 2 years ago
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
Hi, I am facing the same issues with reproducibility for the Longformer model for sequence classification. I posted my example in the Huggingface discussion forum here.
Setting the seeds as recommended here produces the exact same training loss in multiple training iterations (each time starting the finetuning from scratch) when using the roberta-base
model, but not with allenai/longformer-base-4096
.
Hmm, that's interesting! Also pinging the original author @ibeltagy here.
I don't see any weird functionality that's used in Longformer maybe except torch.Tensor.stride(...)
that is in Longformer but not in other models like RoBERTa. Sadly I won't have the time to dive deep into this reproducible problem, but a good start to find the bug would be to verify that the same random model weights are loaded before training and then working with print(...)
to see after what layer results start to differ
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
@abhishekkrthakur do you know where the randomness comes from?
I have recently run into this issue. One of the things I've noticed is if you drop the max length down to 512 I start to get reproducible behaviour - not sure if that helps at all
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
@abhishekkrthakur @patrickvonplaten I'm facing same issue. Any update on this behaviour ?
Environment info transformers Version: 4.11.0 Platform: Amazon Linux AMI 2018.03 torch Version: 1.9.0 Number GPU: 1
more information, when activating torch.use_deterministic_algorithms(True) during training. roberta-base and bert-base-uncased works fine. However with Longformer I get this error:
Loading features from cached file ./training-data/processed-data/text/toy/cached_train_longformer_512
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Validation sanity check: 0it [00:00, ?it/s]Loading features from cached file ./training-data/processed-data/text/toy/cached_dev_longformer_512
Validation sanity check: 0%| | 0/2 [00:00<?, ?it/s]inputs={'input_ids': tensor([[ 0, 7202, 3063, ..., 1, 1, 1],
[ 0, 1301, 1723, ..., 44828, 15555, 2],
[ 0, 27201, 1000, ..., 3706, 6, 2],
...,
[ 0, 41188, 2444, ..., 1, 1, 1],
[ 0, 3384, 591, ..., 1, 1, 1],
[ 0, 495, 2492, ..., 700, 28607, 2]], device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, ..., 0, 0, 0],
[1, 1, 1, ..., 1, 1, 1],
[1, 1, 1, ..., 1, 1, 1],
...,
[1, 1, 1, ..., 0, 0, 0],
[1, 1, 1, ..., 0, 0, 0],
[1, 1, 1, ..., 1, 1, 1]], device='cuda:0'), 'labels': tensor([1, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1,
0, 0, 1, 0, 1, 0, 1, 1], device='cuda:0')}
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/venv-transformer/bin/nlpsaf-transformer-train", line 33, in
Facing the same issue with allenai/led-large-16384 via the run_summarization.py script. Both seed
and data_seed
are set, and there are no randomly initialized weights:
[INFO|modeling_utils.py:3032] 2023-03-30 10:44:26,097 >> All model checkpoint weights were used when initializing LEDForConditionalGeneration.
[INFO|modeling_utils.py:3040] 2023-03-30 10:44:26,098 >> All the weights of LEDForConditionalGeneration were initialized from the model checkpoint at allenai/led-large-16384.
The same script with another model (e.g. flan-t5-large
) is perfectly reproducible across runs with the same seed. I don't have a sense of where the remaining sources of randomness would be for Longformer/LED.
Environment info
transformers
version: 4.8.1Who can help
@patrickvonplaten
Information
Results when using
Longformer
for sequence classification are not consistent across runs after setting a random seed.To reproduce
I include below a simple script derived from this example to reproduce the behavior.
When using
bert-base-uncased
, the generated results (and loss values) will be exactly the same across runs. However, when usingallenai/longformer-base-4096
(just swap the comment for lines starting withMODEL_NAME
), results (and loss values) will vary across runs. In this example, results happen to be very similar because of the simplicity of the problem, but I have experienced higher variability in cases with longer training schedules and larger and more 'complex' datasets. Still, I think this example suffices to illustrate the issue.P.S. Using the commented
set_seed
function does not help.Expected behavior
I would expect that, having set a random seed, results would be the same across runs for longformers too. Is there any source of randomness for longformers not 'covered' by the
set_seed
function?Thanks!