the effectiveness of image detection

[breadxiexie]

Hello, I have some questions about the details related to dire ft in the paper. Is Dire FT using new categories of data (such as bedrooms mentioned in the article) to continue training on the original model? Did it perform poorly in categories that did not undergo such fine-tuning, but showed a significant improvement after fine-tuning? Have you ever conducted an experiment where only one category of data is used for training and verifying the detection performance of images in that category? Another question: As mentioned in the article, the classifier performance of non linear models may not necessarily be better than that of linear classifiers. Is this also due to the limitations of the data categories mentioned above? Can it be assumed that if the same categories are used for training, the detection performance of these methods (such as DIRE) will be better for the same categories?

[breadxiexie]

Also, regarding the CBM mentioned in the paper, is it applied to linear classifiers? How are these features (such as fingers) applied?

[nosna]

Hi,

Is Dire FT using new categories of data (such as bedrooms mentioned in the article) to continue training on the original model? Did it perform poorly in categories that did not undergo such fine-tuning, but showed a significant improvement after fine-tuning?

In our paper, DIRE ZS is the "LSUN-ADM" pre-trained checkpoint that was trained on LSUN-B.(bedrooms images). DIRE FT is trained from scratch with our training set (real news images and fake news images from Midjourney) without seeing any bedroom images. As you can see in Fig 5 of our paper, DIRE ZS performed very poorly (< 10% F-1) across all of our test sets due to the huge domain gap from bedrooms to news images. After training with our dataset, DIRE FT has a huge performance boost (~60% F-1). Since this is trained and tested on news images, so this might be the "experiment where only one category of data is used for training and verifying the detection performance of images in that category" as you mentioned.

As mentioned in the article, the classifier performance of non linear models may not necessarily be better than that of linear classifiers. Is this also due to the limitations of the data categories mentioned above?

Regarding the comparison between the linear model and object-class concept bottleneck model(CBM), we suspect that the lower average performance of CBM is due to the lack of global features (since it's trained on regional features - crops of different objects). However, in Table 5 of our ablation study, the Linear model (second row: MiRAGe-I without CBM) is better at detecting real images while CBM (third row: MiRAGe-I without Linear) is better at detecting fake images.

Can it be assumed that if the same categories are used for training, the detection performance of these methods (such as DIRE) will be better for the same categories?

Generally, testing on the same domain would have higher performance than testing on exclusively different domains (i.e. DIRE FT > DIRE ZS)

Also, regarding the CBM mentioned in the paper, is it applied to linear classifiers? How are these features (such as fingers) applied?

There's a linear model for each object class (300 total, and fingers is one of them), and the output logits from all 300 linear models (a [300,1] vector) are passed through a linear layer for final prediction (0 if the object is not detected). For example, if fingers are detected by Owl V2 during inference, the crop of fingers would be classified as real or fake fingers by the "fingers linear model", and the logits of that model would be used along with other 299 logits to make a final prediction of the whole image.

Let me know if you have further questions

[breadxiexie]

Your careful and meticulous answers have been of great help to me, and I am extremely grateful! After reading the code of DIRE, I found that this method consists of two steps: first, using a diffusion model to reconstruct, and then generating the difference dire between the reconstructed image (recons) and the original image (image). Then, in the second step, a linear classifier is used to train and classify the dirty images (labeled as real or fake). This can also explain why models on a dataset (bedroom) cannot correctly predict other categories. So I speculate whether it is possible to use some fixed metrics (possibly trained to obtain this fixed value) to make accurate predictions for any category of images (after reconstruction).

[breadxiexie]

Also, does CBM contain any semantic information? I have seen working at LOKI recently（ https://opendatalab.github.io/LOKI/ ）I saw the use of labeliu (using multimodal models for annotation) to quickly detect abnormal details in videos or images.