Error when doing white-box attack on OneFormer model

[momo1986]

Hi,

I focused on the problem of AI security.

I want to study the performance of OneFormer model under some white-box attacks, e.g., FGSM.

The script is "python train_net_adv.py --num-gpus 1 --config-file configs/cityscapes/convnext/oneformer_convnext_large_bs16_90k.yaml --eval-only MODEL.IS_TRAIN False MODEL.WEIGHTS 250_16_convnext_l_oneformer_cityscapes_90k.pth MODEL.TEST.TASK semantic".

It only implements on the validation dataset of cityscpaes.

I set the model in the train mode to get the information of gradient in OneFormer model. Then attack the image. Afterward, let OneFormer model inference on the images again.

Here is the error log:

File "train_net.py", line 436, in launch( File "/home/ubuntu/junyan/Python/segmentation/detectron2/detectron2/engine/launch.py", line 84, in launch main_func(args) File "train_net_adv.py", line 417, in main res = Trainer.test(cfg, model) File "train_net_adv.py", line 367, in test results_i = inference_on_dataset(model, data_loader, evaluator) File "/home/ubuntu/junyan/Python/segmentation/detectron2/detectron2/evaluation/evaluator.py", line 261, in inference_on_dataset outputs = model(inputs) File "/home/ubuntu/anaconda3/envs/oneformer/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(input, **kwargs) File "/home/ubuntu/junyan/Python/segmentation/OneFormer/oneformer/oneformer_model.py", line 283, in forward texts = torch.cat([self.text_tokenizer(x["text"]).to(self.device).unsqueeze(0) for x in batched_inputs], dim=0) File "/home/ubuntu/junyan/Python/segmentation/OneFormer/oneformer/oneformer_model.py", line 283, in texts = torch.cat([self.text_tokenizer(x["text"]).to(self.device).unsqueeze(0) for x in batched_inputs], dim=0) KeyError: 'text'

If I inference on the testing/validation dataset, is there a correct way to open the training model of OneFormer and get the internal gradient information?

I wish you are all good.

I am grateful to your contributions and generosity.

Thanks & Regards! Yours Sincerely, Momo

[praeclarumjj3]

Hi @momo1986, the error arises because you probably do not prepare an input text in your dataset mapper. Are you sure you are using the correct dataset mapper?

https://github.com/SHI-Labs/OneFormer/blob/4962ef6a96ffb76a76771bfa3e8b3587f209752b/oneformer/data/dataset_mappers/oneformer_unified_dataset_mapper.py#L372

https://github.com/SHI-Labs/OneFormer/blob/4962ef6a96ffb76a76771bfa3e8b3587f209752b/oneformer/oneformer_model.py#L282-L283

[momo1986]

Hi, @praeclarumjj3.

Thanks for your reply.

I know that the evaluation dataset should be given the token "text" to enable the training mode.

Is there a demo that "text" field could be edited and set in a correct parameter when "--eval-only" is given.

I wish you are all good.

Best Wishes! Yours Sincerely, Momo

[praeclarumjj3]

Hi @momo1986, you would need to create a "text" list inside the evaluation dataset mapper if you want to use it during inference. https://github.com/SHI-Labs/OneFormer/blob/4962ef6a96ffb76a76771bfa3e8b3587f209752b/oneformer/data/dataset_mappers/dataset_mapper.py#L21

You can take a look at the training dataset mapper for reference. https://github.com/SHI-Labs/OneFormer/blob/4962ef6a96ffb76a76771bfa3e8b3587f209752b/oneformer/data/dataset_mappers/oneformer_unified_dataset_mapper.py#L26

[momo1986]

Hi, I have generated the panoptic file as README.md prompted.

I also set as the training mode. python train_net_adv.py --num-gpus 1 --config-file configs/cityscapes/convnext/oneformer_convnext_large_bs16_90k.yaml --eval-only MODEL.IS_TRAIN True MODEL.WEIGHTS 250_16_convnext_l_oneformer_cityscapes_90k.pth MODEL.TEST.TASK semantic However, I obtained the error.

This is the error log.

Traceback (most recent call last): File "train_net_adv.py", line 438, in launch( File "/home/ubuntu/junyan/Python/segmentation/detectron2/detectron2/engine/launch.py", line 84, in launch main_func(args) File "train_net_adv.py", line 419, in main res = Trainer.test(cfg, model) File "train_net_adv.py", line 369, in test results_i = inference_on_dataset(model, data_loader, evaluator) File "/home/ubuntu/junyan/Python/segmentation/detectron2/detectron2/evaluation/evaluator.py", line 241, in inference_on_dataset for idx, inputs in enumerate(data_loader): File "/home/ubuntu/anaconda3/envs/oneformer/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in next data = self._next_data() File "/home/ubuntu/anaconda3/envs/oneformer/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data return self._process_data(data) File "/home/ubuntu/anaconda3/envs/oneformer/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data data.reraise() File "/home/ubuntu/anaconda3/envs/oneformer/lib/python3.8/site-packages/torch/_utils.py", line 434, in reraise raise exception ValueError: Caught ValueError in DataLoader worker process 0. Original Traceback (most recent call last): File "/home/ubuntu/anaconda3/envs/oneformer/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop data = fetcher.fetch(index) File "/home/ubuntu/anaconda3/envs/oneformer/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/ubuntu/anaconda3/envs/oneformer/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/ubuntu/junyan/Python/segmentation/detectron2/detectron2/data/common.py", line 95, in getitem data = self._map_func(self._dataset[cur_idx]) File "/home/ubuntu/junyan/Python/segmentation/detectron2/detectron2/utils/serialize.py", line 26, in call return self._obj(args, **kwargs) File "/home/ubuntu/junyan/Python/segmentation/OneFormer/oneformer/data/dataset_mappers/oneformer_unified_dataset_mapper.py", line 301, in call raise ValueError( ValueError: Cannot find 'pan_seg_file_name' for panoptic segmentation dataset datasets/cityscapes/leftImg8bit/val/frankfurt/frankfurt_000001_062250_leftImg8bit.png.

I feel sorry for disturing you. Nevertheless, I am looking forward to addressing the problem.

Thanks & Regards! Yours Sincerely, Momo

[momo1986]

Looks the pan seg file is existed, how can I link them together?

The file structure is set as recommended.

Thanks & Best wishes! Regards! Momo

[momo1986]

Looks like that the model would touch the task of panoptic segmentation when the mode is set to the training model. However, for attacking adversarially in the inference time, evaluation only on semantic segmentation is what we want? Is there any workaround in UI to skip this issue? Thanks & Regards！ Momo

[praeclarumjj3]

Hi @momo1986, were you able to solve this issue? I am sorry for not being able to reply earlier, been busy.

[praeclarumjj3]

I am closing this issue due to inactivity. Feel free to re-open.