search

lucidrains / vit-pytorch

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch
MIT License
19.52k stars 2.95k forks source link

issue VIT + Loss function #214

Open jeremyEudaric opened 2 years ago

jeremyEudaric commented 2 years ago

Hello, l am using VIT on images with 2 classes. ['horses', 'humans']

model= ViT( image_size = 256, patch_size = 32, num_classes = 2, dim = 1024, depth = 6, heads = 16, mlp_dim = 2048, dropout = 0.1, emb_dropout = 0.1 )

However l can not understand why my loss function jump and the accuracy stay around 0.52 is not a data issue because l have the problem on several data set. l am studied cancer cells but , l tested VIT on random data before :

Thank you for your helps any advices ?

Epoch : 7 - loss : 29.5589 - acc: 0.5694 - val_loss : 33.4046 - val_acc: 0.4931

100%|█████████████████████████████████████████████| 6/6 [00:09<00:00, 1.55s/it] 0%| | 0/6 [00:00<?, ?it/s]

Epoch : 8 - loss : 32.3972 - acc: 0.5243 - val_loss : 40.7422 - val_acc: 0.5000

100%|█████████████████████████████████████████████| 6/6 [00:09<00:00, 1.56s/it] 0%| | 0/6 [00:00<?, ?it/s]

Epoch : 9 - loss : 42.9368 - acc: 0.4201 - val_loss : 46.1266 - val_acc: 0.5069

100%|█████████████████████████████████████████████| 6/6 [00:09<00:00, 1.52s/it] 0%| | 0/6 [00:00<?, ?it/s]

Epoch : 10 - loss : 45.1841 - acc: 0.4792 - val_loss : 35.0454 - val_acc: 0.4792

100%|█████████████████████████████████████████████| 6/6 [00:08<00:00, 1.48s/it] 0%| | 0/6 [00:00<?, ?it/s]

Epoch : 11 - loss : 32.7426 - acc: 0.4965 - val_loss : 38.8612 - val_acc: 0.4792

100%|█████████████████████████████████████████████| 6/6 [00:11<00:00, 1.87s/it] 0%| | 0/6 [00:00<?, ?it/s]

Epoch : 12 - loss : 37.5920 - acc: 0.4757 - val_loss : 73.4655 - val_acc: 0.5208

100%|█████████████████████████████████████████████| 6/6 [00:09<00:00, 1.50s/it] 0%| | 0/6 [00:00<?, ?it/s]

Epoch : 13 - loss : 87.3413 - acc: 0.5139 - val_loss : 30.2588 - val_acc: 0.4931

100%|█████████████████████████████████████████████| 6/6 [00:08<00:00, 1.49s/it] 0%| | 0/6 [00:00<?, ?it/s]

Epoch : 14 - loss : 30.0461 - acc: 0.5312 - val_loss : 29.3259 - val_acc: 0.5486

100%|█████████████████████████████████████████████| 6/6 [00:09<00:00, 1.51s/it]

Epoch : 15 - loss : 29.9363 - acc: 0.5243 - val_loss : 44.0025 - val_acc: 0.5069

cuteboyqq commented 1 month ago

l encounter the same issue, l got low acc when train ViT model , dataset using CIFAR10, CIFAR100, imagenet100, all acc is vary low, does anyone know what is going on ?