Bug in anomaly_detection.py

[zhiyuanyou]

Hi~

The GDE model should be built with pure embeddings of normal samples. However, in anomaly_detection.py, you build GDE model with embeddings of training samples. Some of these training samples have been transformed by CutPaste, which should be considered as anomalies. So I wonder whether there are some problems.

[zhiyuanyou]

Also, when I revise your code, build GDE model with embeddings of training samples that have not been transformed by CutPaste, the performance is obviously improved.

[RuojiWang]

well, that's interesting. i may try "build GDE model with embeddings of training samples that have not been transformed by CutPaste" and i am wonder whether it can improve my result.

[Chichiviriche]

Hi 1368069096, can you share your modifications to build GDE model with pure embeddings of normal samples ? Thanks in advance

[zhiyuanyou]

Hi, sorry for missing this message. My revisions are shown as follows.

1. get embeddings of normal samples.
2. fit a GDE model with normal embeddings.

class AnomalyDetection:

    def create_embeds_normal(self, meta_file) -> Tuple[torch.Tensor, torch.Tensor]:
        """
        load trainset with embeds and labels for tsne
            labels in [0, 1, 2] when using 3way or [0, 1], label==0 means "good" 
        Returns:
            Tuple[torch.Tensor, torch.Tensor]: embeds, labels
        """
        embeddings = []
        dataset = MVTecAD(meta_file=meta_file, root_dir="", mode='train_no_cutpaste')
        dataloader = DataLoader(dataset, batch_size = self.batch_size)
        with torch.no_grad():
            for imgs in dataloader:
                logits, embeds = self.cutpaste_model(imgs.to(self.device))
                del logits
                embeddings.append(embeds.to('cpu'))
                torch.cuda.empty_cache()
        return torch.cat(embeddings)

    @staticmethod
    def GDE_fit(train_embeds, save_path = None):
        GDE = KernelDensity().fit(train_embeds)
        if save_path:
            filename = os.path.join(save_path, 'GDE.sav')
            pickle.dump(GDE, open(filename, 'wb'))
        return GDE

    def mvtec_anomaly_detection(self, meta_file, data_path, save_path=None):
        # get embeddings of normal samples
        train_embeds = self.create_embeds_normal(meta_file)
        # fit a GDE model with normal embeddings
        GDE_model = self.GDE_fit(train_embeds, save_path)

[Chichiviriche]

Hello zhiyuanyou, thanks a lot for sharing those modifications. In the meantime, I've also modified more or less in the same way and get better results By the way, the next difficulty in to get localization.py working. Did you succeed ? Modifying what ? Kind regards,

[zhiyuanyou]

Hi~

Actually no. The reason is that the inference time for localization is too long. We finally only used CurPaste as our anomaly detection baseline. 

You could see our paper for more results. 

paper: A Unified Model for Multi-class Anomaly Detection，Accepted by NeurIPS 2022. Code: https://github.com/zhiyuanyou/UniAD

---Original--- From: @.> Date: Wed, Oct 5, 2022 15:13 PM To: @.>; Cc: @.**@.>; Subject: Re: [LilitYolyan/CutPaste] Bug in anomaly_detection.py (Issue #24)

Hello zhiyuanyou, thanks a lot for sharing those modifications. In the meantime, I've also modified more or less in the same way and get better results By the way, the next difficulty in to get localization.py working. Did you succeed ? Modifying what ? Kind regards,

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[Chichiviriche]

Thanks, interesting, I will look into detail. Have a good day,

[Chichiviriche]

Sorry to bother you again, in your modifications above, you have lines such ;

• def mvtec_anomaly_detection(self, meta_file, data_path, save_path=None): or
• dataset = MVTecAD(meta_file=meta_file, root_dir="", mode='train_no_cutpaste') that indicate that you have other modifications that what you provided. Can you explain ? Thanks,

[zhiyuanyou]

Hi~

We revised the dataset part to get normal embeddings.

In anomaly_detection.py: dataset = MVTecAD(meta_file=meta_file, root_dir="", mode='train_no_cutpaste')

Then , the "MVTecAD" class is defined in dataset.py as:

class MVTecAD(Dataset):
    def __init__(
        self, 
        meta_file,
        root_dir,
        mode,
        image_size = (256,256), 
        cutpaste_type = '3way',
        localization = False
        ):

        self.mode = mode
        self.meta_file = meta_file
        self.root_dir = root_dir
        self.cutpaste_transform = CutPaste(type=cutpaste_type)

        self.crop_size = (64,64) if localization else image_size

        self.transform = transforms.Compose([transforms.Resize(image_size),
                                             transforms.RandomCrop(self.crop_size),
                                             transforms.ToTensor(),
                                             transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
                                             ])

        self.mask_transform = transforms.Compose([transforms.Resize(image_size, transforms.InterpolationMode.NEAREST),
                                             transforms.RandomCrop(self.crop_size),
                                             transforms.ToTensor(),
                                             ])

        if self.mode == "train" or self.mode == "train_no_cutpaste":
            images = []
            with open(meta_file) as fr:
                for line in fr:
                    meta = json.loads(line)
                    images.append(os.path.join(self.root_dir, meta["filename"]))
            self.images = images
        elif self.mode == 'test':
            images = []
            labels = []
            masks = []
            with open(meta_file) as fr:
                for line in fr:
                    meta = json.loads(line)
                    images.append(os.path.join(self.root_dir, meta["filename"]))
                    labels.append(meta["label"])
                    if meta.get("maskname", None):
                        masks.append(os.path.join(self.root_dir, meta["maskname"]))
                    else:
                        masks.append(None)
            self.images = images
            self.labels = labels
            self.masks = masks
        else:
            raise ValueError('Please specify dataset path and mode')

    def __len__(self):
        return len(self.images)

    def __getitem__(self, item):
        """
        :return: torch tensor if mode is 'train' else torch tensor and label
        """
        if self.mode == "train":
            image_path = self.images[item]
            image = Image.open(image_path).convert('RGB')
            out = self.cutpaste_transform(image)
            transformed = [self.transform(i) for i in out]
            return transformed
        elif self.mode == "train_no_cutpaste":
            image_path = self.images[item]
            image = Image.open(image_path).convert('RGB')
            image = self.transform(image)
            return image
        elif self.mode == "test":
            image_path = self.images[item]
            mask = self.masks[item]
            label = self.labels[item]
            image = Image.open(image_path).convert('RGB')
            if mask:
                mask = Image.open(mask)
            else:
                assert label == 0
                mask = Image.fromarray(np.zeros(self.crop_size))
            image = self.transform(image)
            mask = self.mask_transform(mask)
            return image, mask, label
        else:
            raise ValueError('Please specify dataset path and mode')

Also, the meta_file should be prepared as:

{"filename": image path, "label": 0 for normal and 1 for anomaly, "maskname": mask path for anomaly, None for normal}
{"filename": image path, "label": 0 for normal and 1 for anomaly, "maskname": mask path for anomaly, None for normal}
{"filename": image path, "label": 0 for normal and 1 for anomaly, "maskname": mask path for anomaly, None for normal}
......

[Chichiviriche]

Sorry to bother you again, in your modifications above, I'm not familiar with meta_file. 1) How do you prepare the meta_file to be used in def mvtec_anomaly_detection(self, meta_file, data_path, save_path=None) ? 2) I guess that you also modify train.py to fit with you revised dataset.py Thanks a lot in advance !