Accuracy not as per paper

bishshoy commented 5 years ago

For the following code:

import torch, torchvision
from efficientnet_pytorch import EfficientNet
from tqdm import tqdm

def main(l):
    model = EfficientNet.from_pretrained(model_name='efficientnet-b'+str(l))
    model.cuda()
    model.eval()
    dataset = torchvision.datasets.ImageNet('/home/milton/.torch/datasets/', split='val', download=False,
                                                               transform=torchvision.transforms.Compose([
                                                                   torchvision.transforms.Resize(256),
                                                                   torchvision.transforms.CenterCrop(224),
                                                                   torchvision.transforms.ToTensor(),
                                                                   torchvision.transforms.Normalize((0.485, 0.456, 0.406),
                                                                                                    (0.229, 0.224, 0.225)),
                                                               ]))
    dataloader = torch.utils.data.DataLoader(dataset, batch_size=200, shuffle=False, num_workers=40)

    with torch.no_grad():
        correct_predictions = torch.zeros(1)
        correct_topk_predictions = torch.zeros(1)
        for data, labels in tqdm(dataloader):
            data, labels = data.cuda(), labels.cuda()
            output = model(data)
            predictions = output.argmax(dim=1, keepdim=True)
            correct_predictions += predictions.eq(labels.view_as(predictions)).sum()
            _, predictions_topk = output.topk(5, 1, True, True)
            predictions_topk = predictions_topk.t()
            correct_topk_predictions += predictions_topk.eq(labels.view(1, -1).expand_as(predictions_topk)).sum()

    print(100*(correct_predictions/dataset.__len__()).item(), 100*(correct_topk_predictions/dataset.__len__()).item())

if __name__ == '__main__':
    for l in range(8):
        main(l)

I am getting the following accuracies:

Loaded pretrained weights for efficientnet-b0
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [00:57<00:00,  5.18it/s]
76.1240005493164 92.97999739646912
100%|█████████████████████████████████████████████████████████████████████████████| 31519111/31519111 [00:02<00:00, 11364441.82it/s]
Loaded pretrained weights for efficientnet-b1
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [01:10<00:00,  3.96it/s]
77.88599729537964 93.76800060272217
100%|██████████████████████████████████████████████████████████████████████████████| 36804509/36804509 [00:04<00:00, 8495839.83it/s]
Loaded pretrained weights for efficientnet-b2
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [01:13<00:00,  3.77it/s]
77.94600129127502 93.76999735832214
100%|██████████████████████████████████████████████████████████████████████████████| 49388949/49388949 [00:05<00:00, 9453856.32it/s]
Loaded pretrained weights for efficientnet-b3
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [01:31<00:00,  2.95it/s]
77.78000235557556 93.60799789428711
100%|█████████████████████████████████████████████████████████████████████████████| 77999237/77999237 [00:06<00:00, 12893608.18it/s]
Loaded pretrained weights for efficientnet-b4
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [01:57<00:00,  2.27it/s]
77.69399881362915 93.57600212097168
100%|███████████████████████████████████████████████████████████████████████████| 122410125/122410125 [00:09<00:00, 13483970.56it/s]
Loaded pretrained weights for efficientnet-b5
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [02:36<00:00,  1.68it/s]
75.95000267028809 92.69000291824341
100%|███████████████████████████████████████████████████████████████████████████| 173245619/173245619 [00:13<00:00, 12484517.39it/s]
Loaded pretrained weights for efficientnet-b6
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [03:19<00:00,  1.30it/s]
76.528000831604 93.0679976940155
Loaded pretrained weights for efficientnet-b7
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [04:24<00:00,  1.03s/it]
76.83600187301636 93.21600198745728

The accuracies are not as per the paper. Am I doing something wrong? Please help me. Sorry for being a noob.

shijianjian commented 5 years ago

Could you upload your environment settings as well? Say, the versioning of PyTorch, CUDA, etc.

bishshoy commented 5 years ago

PyTorch version: 1.1.0 Cuda version: 10.1 Nvidia Driver version: 418.56 Python version: 3.7.4 GPU: GTX 2080 Ti

Anything else that I should add?

shijianjian commented 5 years ago

I found CUDA 10.1 performs weirdly on my testing as well with EfficientNet referring to https://github.com/pytorch/pytorch/issues/28197. I suggest giving a try on reverting the CUDA back from 10.1 to 10.0. Please let me if any luck or the problem stills.

bishshoy commented 5 years ago

What version of cuda, pytorch, Nvidia-driver and cudnn are you using?

I tried with the latest version of PyTorch v1.3.0 (nightly) with cuda v10.1.168 and driver 430.50, I'm still getting the same results.

Downloading: "http://storage.googleapis.com/public-models/efficientnet/efficientnet-b0-355c32eb.pth" to /home/bishshoy/.cache/torch/checkpoints/efficientnet-b0-355c32eb.pth
100%|███████████████████████████████████████████████████████████████████████████████████████████| 20.4M/20.4M [00:22<00:00, 966kB/s]
Loaded pretrained weights for efficientnet-b0
EfficientNet-B0
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 782/782 [01:37<00:00,  8.03it/s]
76.12199783325195 92.97999739646912
Downloading: "http://storage.googleapis.com/public-models/efficientnet/efficientnet-b1-f1951068.pth" to /home/bishshoy/.cache/torch/checkpoints/efficientnet-b1-f1951068.pth
100%|██████████████████████████████████████████████████████████████████████████████████████████| 30.1M/30.1M [00:15<00:00, 2.05MB/s]
Loaded pretrained weights for efficientnet-b1
EfficientNet-B1
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 782/782 [02:14<00:00,  5.82it/s]
77.88599729537964 93.76800060272217
Downloading: "http://storage.googleapis.com/public-models/efficientnet/efficientnet-b2-8bb594d6.pth" to /home/bishshoy/.cache/torch/checkpoints/efficientnet-b2-8bb594d6.pth
100%|██████████████████████████████████████████████████████████████████████████████████████████| 35.1M/35.1M [00:03<00:00, 11.1MB/s]
Loaded pretrained weights for efficientnet-b2
EfficientNet-B2
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 782/782 [02:23<00:00,  5.45it/s]
77.94600129127502 93.76999735832214
Downloading: "http://storage.googleapis.com/public-models/efficientnet/efficientnet-b3-5fb5a3c3.pth" to /home/bishshoy/.cache/torch/checkpoints/efficientnet-b3-5fb5a3c3.pth
100%|██████████████████████████████████████████████████████████████████████████████████████████| 47.1M/47.1M [00:05<00:00, 9.14MB/s]
Loaded pretrained weights for efficientnet-b3
EfficientNet-B3
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 782/782 [03:08<00:00,  4.15it/s]
77.78000235557556 93.60799789428711
Downloading: "http://storage.googleapis.com/public-models/efficientnet/efficientnet-b4-6ed6700e.pth" to /home/bishshoy/.cache/torch/checkpoints/efficientnet-b4-6ed6700e.pth
100%|██████████████████████████████████████████████████████████████████████████████████████████| 74.4M/74.4M [00:09<00:00, 8.12MB/s]
Loaded pretrained weights for efficientnet-b4
EfficientNet-B4
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 782/782 [04:12<00:00,  3.10it/s]
77.69399881362915 93.57600212097168
Downloading: "http://storage.googleapis.com/public-models/efficientnet/efficientnet-b5-b6417697.pth" to /home/bishshoy/.cache/torch/checkpoints/efficientnet-b5-b6417697.pth
100%|████████████████████████████████████████████████████████████████████████████████████████████| 117M/117M [00:10<00:00, 11.7MB/s]
Loaded pretrained weights for efficientnet-b5
EfficientNet-B5
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 782/782 [05:47<00:00,  2.88it/s]
75.95000267028809 92.69000291824341
Downloading: "http://storage.googleapis.com/public-models/efficientnet/efficientnet-b6-c76e70fd.pth" to /home/bishshoy/.cache/torch/checkpoints/efficientnet-b6-c76e70fd.pth
100%|████████████████████████████████████████████████████████████████████████████████████████████| 165M/165M [00:17<00:00, 9.89MB/s]
Loaded pretrained weights for efficientnet-b6
EfficientNet-B6
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 782/782 [07:34<00:00,  2.22it/s]
76.528000831604 93.0679976940155
Downloading: "http://storage.googleapis.com/public-models/efficientnet/efficientnet-b7-dcc49843.pth" to /home/bishshoy/.cache/torch/checkpoints/efficientnet-b7-dcc49843.pth
100%|████████████████████████████████████████████████████████████████████████████████████████████| 254M/254M [00:22<00:00, 11.7MB/s]
Loaded pretrained weights for efficientnet-b7
EfficientNet-B7
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 782/782 [09:43<00:00,  1.72it/s]
76.83600187301636 93.21600198745728

drcdr commented 5 years ago

I'm new to Efficient-Net. Using your script, I get the same results. From looking at examples/imagenet/main.py as well as #12 - image-size and interpolation are important in EfficientNets. A modified version of your script to incorporate this is:

import torch, torchvision
from efficientnet_pytorch import EfficientNet
from tqdm import tqdm
import numpy, matplotlib
import PIL

torch.backends.cudnn.benchmark = True
torch.backends.cudnn.enabled = True

def main(l):
    path_to_imagenet = YOUR-PATH-HERE
    arch = 'efficientnet-b'+str(l)
    model = EfficientNet.from_pretrained(model_name=arch)
    image_size = EfficientNet.get_image_size(arch)
    batch_size = 32  # 200 for 0, but ~32 for 7
    model.cuda()
    model.eval()
    transform = torchvision.transforms.Compose([
        torchvision.transforms.Resize(image_size, interpolation=PIL.Image.BICUBIC), # BILINEAR
        torchvision.transforms.CenterCrop(image_size),
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                         std=[0.229, 0.224, 0.225]),
    ])
    dataset = torchvision.datasets.ImageNet(
        path_to_imagenet, split='val', download=False, transform=transform)
    dataloader = torch.utils.data.DataLoader(
        dataset, batch_size=batch_size, shuffle=False, num_workers=4)

    #get_model_flops(model, dataset)

    with torch.no_grad():
        correct_predictions = torch.zeros(1)
        correct_topk_predictions = torch.zeros(1)
        for data, labels in tqdm(dataloader):
            data, labels = data.cuda(), labels.cuda()
            output = model(data)
            predictions = output.argmax(dim=1, keepdim=True)
            correct_predictions += predictions.eq(labels.view_as(predictions)).sum()
            _, predictions_topk = output.topk(5, 1, True, True)
            predictions_topk = predictions_topk.t()
            correct_topk_predictions += predictions_topk.eq(
                labels.view(1, -1).expand_as(predictions_topk)).sum()

    print('Top1: %9.6f  Top5: %9.6f (imsz=%d)' %
          (100*(correct_predictions/dataset.__len__()).item(),
           100*(correct_topk_predictions/dataset.__len__()).item(), image_size))

if __name__ == '__main__':
    print_versions()
    for l in range(8):
        main(l)

See some results in the table below.

Columns E...F are the new Tensorflow-EfficientNet AutoAugment results, just for reference
Columns G...J are your results
Columns K...O is the code above and should be equivalent to running examples/imagenet/main.py; scaling images using BICUBIC to a larger size.
Columns P...T are just a variation, seeing how much worse BILINEAR is than BICUBIC.
(1,5,d1,d5) == (Top1Val, Top5Val, Top1Val vs. Paper, Top5Val vs. Paper)

I also tried counting the model flops, using both thop and ptflops. They gave very low estimates, off by a factor of 30-50 at the high end. ptflops did get the number of params correct, just not the flops; maybe it's because of the use of F.conv2d instead of torch.nn.Conv2d.

Here are some possible next-steps:

convert these to ONNX, and see if the results remain the same
make get_model_flops() work properly?
can the new AutoAugment weights could be pulled into Efficientnet-PyTorch as well? (place to modify)

mathpopo commented 4 years ago

@bishshoy the pths now can download?can you share them here?

EvgeniaAR commented 4 years ago

@drcdr: Do you understand whether the provided weights correspond to the 'Baseline preprocessing' or to the 'AutoAugment' preprocessing from Tensorflow?

drcdr commented 4 years ago

Just comparing the Top-1 accuracies on the EfficientNet-Pytorch homepage (76.3, 78.8, ...,), it looks like the weights are based on the baseline weights here.

I'm hoping https://github.com/pytorch/pytorch/issues/18631 gets fixed soon, but it doesn't look like it made 1.5: https://github.com/pytorch/pytorch/issues/35009

EvgeniaAR commented 4 years ago

That's what I thought as well. But this comment in ReadMe confuses me:

Update (July 31, 2019) Upgrade the pip package with pip install --upgrade efficientnet-pytorch

The B6 and B7 models are now available. Additionally, all pretrained models have been updated to use AutoAugment preprocessing, which translates to better performance across the board. Usage is the same as before:

drcdr commented 4 years ago

Ah, I missed that! A little confusing, but I guess it's AutoAugment then.

lukemelas commented 4 years ago

Hi all! Sorry for the confusion about the checkpoints. They are indeed the AutoAugment checkpoints. I decided to replace the baseline checkpoints with the AA ones because the AA checkpoints are essentially better versions of the baseline checkpoints (i.e. they're trained in the same way). The advprop checkpoints are trained differently, so I didn't want people to accidentally use the advprop checkpoints without realizing.

As @drcdr said, we're all waiting on pytorch/pytorch#18631 to speed up the convolutions.

EvgeniaAR commented 4 years ago

@lukemelas, thanks for the quick response and this repo! We would like to compare vanilla trained efficient nets to the autoaugment ones on some tasks. If you happen to still have the pytorch vanilla weights somewhere, could you please post a download link? Otherwise, we could probably convert them from tensorflow with you code. Thanks again :).

lukemelas / EfficientNet-PyTorch

Accuracy not as per paper #92