patch training fails to produce useful pattern

[mkrupczak3]

Currently, patch training is failing to generate an adversarial pattern. It's just optimizing for non-printability score (nps) by generating an image of a single color:

@SlicedBacon and I believe this is because the Airbus dataset has negative training images (without objects present) which interfere with the patch training's stochastic gradient descent.

The original paper (and thus, the code repo this one is based on) used the INRIA dataset for person detection. @SlicedBacon noted this dataset does not have any negative training images where a person is not present.

In the current patch training code, any images without objects present will generate a det_loss of 0.0, giving the optimizer a false signal that the patch completely defeated the object detector. This seems to be interfering with the patch training process, causing the discontinuities observed in the det_loss graph from tensorboard shown above.

For a quick fix, we may remove the negative training images from the dataset when using it for patch training. We should fix it properly so it ignores negative training images in the future though so other researchers can use our code

[mkrupczak3]

Removing the negative training images seems to have had little impact. Same result as before.

At our meeting today, discussed how this might be because the patch training dataset is much too large. Theory is that with so many different images, the model is having a hard time learning what works and what doesn't with any single change to the patch.

Going to try a training run on a much smaller subset of data (about 8 images) and if it does any better

[mkrupczak3]

Here was the result after 5000 epochs on a set of 8 images:

Not enough time in the oven. Doing another run over the weekend, this time of 500,000 epochs to see if it works

[dkyman]

Hello, I would like to ask why the patch generated by running the code has no gradient. It means that the patch has no changes after multiple rounds of training.I am using the pedestrian data set and the original patch is gray。

[mkrupczak3]

It's a known issue, the way we are dealing with the tensors is a bit flawed. There's some significant architectural differences between YOLOv4 and YOLOv8 we haven't been able to work out yet

[mkrupczak3]

Here was the result after 5000 epochs on a set of 8 images:

Not enough time in the oven. Doing another run over the weekend, this time of 500,000 epochs to see if it works

This didn't do any better; issue is something with the tensors

[starktao]

So, does this mean that your project cannot successfully train adversarial examples？

[mkrupczak3]

At present, yes. @SlicedBacon has been making some progress but I don't think it's been published here yet

[starktao]

That's unfortunate, but thank you.