About environment setting

[MarkDeng1]

Hi authors,

I met an environment setting issue, when i run eval_PromptStealer.py file.

I notice that your torch version required is very outdated (pytorch 1.12.0a0+8a1a93a)， which is unavailable right now.

The error i encountered, when running eval_PromptStealer.py file:

Traceback (most recent call last): File "eval_PromptStealer.py", line 199, in main() File "eval_PromptStealer.py", line 194, in main evaluate_prompt_stealer(prompt_stealer, val_loader, save_path) File "eval_PromptStealer.py", line 156, in evaluate_prompt_stealer inferred_prompts, pred_modifiers = prompt_stealer.infer(images, lexica_dataset=val_loader.dataset) File "eval_PromptStealer.py", line 69, in infer modifiers = self.infer_modifiers(images, lexica_dataset) File "eval_PromptStealer.py", line 85, in infer_modifiers output_regular = Sig(self.modifier_detector(images.to(device))).cpu() File "/fred/oz337/zdeng/promptStealing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/fred/oz337/zdeng/promptStealing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, *kwargs) File "/fred/oz337/zdeng/prompt-stealing-attack-image/src/ml_decoder/models/tresnet/tresnet.py", line 205, in forward logits = self.head(self.embeddings) File "/fred/oz337/zdeng/promptStealing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/fred/oz337/zdeng/promptStealing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, kwargs) File "/fred/oz337/zdeng/prompt-stealing-attack-image/src/ml_decoder/ml_decoder/ml_decoder.py", line 167, in forward h = self.decoder(tgt, embedding_spatial_786.transpose(0, 1)) # [embed_len_decoder, batch, 768] File "/fred/oz337/zdeng/promptStealing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/fred/oz337/zdeng/promptStealing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(args, kwargs) File "/fred/oz337/zdeng/promptStealing/lib/python3.8/site-packages/torch/nn/modules/transformer.py", line 491, in forward seq_len = _get_seq_len(tgt, self.layers[0].self_attn.batch_first) File "/fred/oz337/zdeng/promptStealing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1729, in getattr raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'") AttributeError: 'TransformerDecoderLayerOptimal' object has no attribute 'self_attn'

How do i deal with this error?

Thanks, Mark

[MarkDeng1]

Additionally, provided modifier.pth is mismatching with tresnet_l model:

Traceback (most recent call last): File "eval_PromptStealer.py", line 199, in main() File "eval_PromptStealer.py", line 192, in main prompt_stealer = PromptStealer(subject_generator_path, modifier_detector_path, device) File "eval_PromptStealer.py", line 29, in init self.load_modifier_detector(modifier_detector_path) File "eval_PromptStealer.py", line 56, in load_modifier_detector self.modifier_detector.load_state_dict(ckpt['model'], strict=True) File "/fred/oz337/zdeng/promptStealing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 2215, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for TResNet: Missing key(s) in state_dict: "head.decoder.layers.0.self_attn.in_proj_weight", "head.decoder.layers.0.self_attn.in_proj_bias", "head.decoder.layers.0.self_attn.out_proj.weight", "head.decoder.layers.0.self_attn.out_proj.bias".

[verazuo]

Hi Mark,

We require this specific PyTorch version because we need to use BLIP, ML_Decoder, and CLIP to train/test PromptStealer. This PyTorch version is compatible with the environment requirements specified by all three packages, so we recommend using it.

As mentioned in our README, NVIDIA provides a Docker environment that supports this setup, which you can refer to here: https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel_22-05.html

The size mismatch issue may also be caused by an incorrectly configured environment.

Best, Vera