Closed ChintanTrivedi closed 2 years ago
vturrisi
commented
2 years ago Hey, which script are you using? I just noticed that typing for that class is wrong, it should be def __call__(self, x: Image) -> Image: but it wouldn't disrupt actually using it.
ChintanTrivedi
commented
2 years ago Like you mentioned, changing the typing did not make any difference, still seeing the same error message AttributeError: 'Tensor' object has no attribute 'filter'. I used this script:-
python3 main_pretrain.py \
--dataset custom \
--backbone resnet18 \
--data_dir <path> \
--train_dir <images_directory> \
--no_labels \
--max_epochs 20 \
--gpus 0 \
--accelerator gpu \
--precision 16 \
--optimizer sgd \
--lars \
--grad_clip_lars \
--eta_lars 0.02 \
--exclude_bias_n_norm \
--scheduler warmup_cosine \
--lr 0.4 \
--classifier_lr 0.1 \
--weight_decay 1e-5 \
--batch_size 32 \
--num_workers 4 \
--crop_size 224 \
--brightness 0.8 \
--contrast 0.8 \
--saturation 0.8 \
--hue 0.2 \
--gaussian_prob 0.0 0.0 \
--num_crops_per_aug 1 1 \
--name <model_name> \
--project solo-learn \
--entity <wandb_name> \
--wandb \
--save_checkpoint \
--method simclr \
--temperature 0.2 \
--proj_hidden_dim 1024 \
--proj_output_dim 128Surprisingly, the dataloader doesn't throw this error when I train a bigger ResNet50 model (instead of 18). I see no relation between the model size and dataloader error, unless it has something to do with the training_step execution time and dataloader queue?
vturrisi
commented
2 years ago Yes. It wasn't supposed to fix your error, just to make the annotations correct.
The issue is that somehow a tensor is being fed to the augmentations pipeline instead of the actual image.
Can you print what is img before? https://github.com/vturrisi/solo-learn/blob/b4fbf788a59ede53631f301a7ba9285602948df9/solo/utils/pretrain_dataloader.py#L94
It would help me to debug.
vturrisi
commented
2 years ago Any update on this?
ChintanTrivedi
commented
2 years ago Sorry for the delay. I had to switch my experiments to ResNet50 and I have not encountered the issue with that setting. While the issue remains unsolved for ResNet18, I am not currently planning to work on that due to time constraints. I shall reopen the issue if I need to use it again. Thanks!
While training a simclr model with custom unlabled data, I am using the default data augmentations. However, randomly, the training would stop, throwing a data-fetch error because of some issue in the data augmentation step:-
Note that the training loop works well for multiple epochs (this example crashed after 9 successful epochs). What could be the issue here? How could I narrow down if this issue is with
pytorch_lightning,torchvisionorsolo-learn?