CUDA error: an illegal memory access was encountered

[buthi]

from tqdm import tqdm from textattack.loggers import CSVLogger from textattack.attack_results import SuccessfulAttackResult from textattack import Attacker from textattack import AttackArgs from textattack.datasets import Dataset

attack_args = AttackArgs(num_examples=10)

recipe.transformation.language = '1'

attacker = Attacker(attack, dataset, attack_args)

attack_results = attacker.attack_dataset()

---

RuntimeError Traceback (most recent call last) /tmp/ipykernel_2819940/43225739.py in 12 attacker = Attacker(attack, dataset, attack_args) 13 ---> 14 attack_results = attacker.attack_dataset()

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/attacker.py in attack_dataset(self) 420 ) 421 --> 422 textattack.shared.utils.set_seed(self.attack_args.random_seed) 423 if self.dataset.shuffled and self.attack_args.checkpoint_interval: 424 # Not allowed b/c we cannot recover order of shuffled data

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/shared/utils/misc.py in set_seed(random_seed) 111 random.seed(random_seed) 112 np.random.seed(random_seed) --> 113 torch.manual_seed(random_seed) 114 torch.cuda.manual_seed(random_seed) 115

~/anaconda3/envs/textattack/lib/python3.7/site-packages/torch/random.py in manual_seed(seed) 38 39 if not torch.cuda._is_in_bad_fork(): ---> 40 torch.cuda.manual_seed_all(seed) 41 42 return default_generator.manual_seed(seed)

~/anaconda3/envs/textattack/lib/python3.7/site-packages/torch/cuda/random.py in manual_seed_all(seed) 111 default_generator.manual_seed(seed) 112 --> 113 _lazy_call(cb, seed_all=True) 114 115

~/anaconda3/envs/textattack/lib/python3.7/site-packages/torch/cuda/init.py in _lazy_call(callable, kwargs) 153 def _lazy_call(callable, kwargs): 154 if is_initialized(): --> 155 callable() 156 else: 157 # TODO(torch_deploy): this accesses linecache, which attempts to read the

~/anaconda3/envs/textattack/lib/python3.7/site-packages/torch/cuda/random.py in cb() 109 for i in range(device_count()): 110 default_generator = torch.cuda.default_generators[i] --> 111 default_generator.manual_seed(seed) 112 113 _lazy_call(cb, seed_all=True)

RuntimeError: CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

[jxmorris12]

Next time, please follow the issue template, and please include the full code for your issue to be reproducible.

Here, torch is failing on torch.cuda.manual_seed(42). Your installation of torch is probably not compatible with your CUDA version. Follow the PyTorch installation instructions here for your system and CUDA version: https://pytorch.org/get-started/locally/

Then, make sure the call to torch.cuda.manual_seed works before trying to run an attack through TextAttack.

[buthi]

下次，请按照问题模板进行操作，并包含完整的代码以使您的问题可重现。

在这里，火炬失败了torch.cuda.manual_seed(42)。您安装的 torch 可能与您的 CUDA 版本不兼容。按照此处针对您的系统和 CUDA 版本的 PyTorch 安装说明进行操作：https ://pytorch.org/get-started/locally/

torch.cuda.manual_seed然后，在尝试通过 TextAttack 运行攻击之前，确保调用有效。

Thank you for your reply. Actually I don't think the problem is that CUDA doesn't match the Torch version. My CUDA version: 11.5 PyTorch: 1.11.0 Py3.7_CUDa11.5_CUDN8.3.2_0. The code in question sometimes works and generates adversarial samples. Sometimes errors are reported.

[buthi]

I uploaded the code to Colab:https://colab.research.google.com/drive/1Frpn8EW3-lFGBqM-_hHtnv67xCDZ5QU0?usp=sharing

[buthi]

The same code sometimes has the following error: textattack: Logging to text file at path /home/ysc/TextAttack/examples/results/PWWS/2022-06-10-19-49-log.txt Attack( (search_method): GreedyWordSwapWIR( (wir_method): weighted-saliency ) (goal_function): UntargetedClassification (transformation): WordSwapWordNet (constraints): (0): RepeatModification (1): StopwordModification (is_black_box): True )

0%| | 0/10 [00:00<?, ?it/s]Building prefix dict from the default dictionary ... Building prefix dict from the default dictionary ... Loading model from cache /tmp/jieba.cache Loading model from cache /tmp/jieba.cache Loading model cost 0.489 seconds. Loading model cost 0.489 seconds. Prefix dict has been built successfully. Prefix dict has been built successfully.

RuntimeError Traceback (most recent call last) /tmp/ipykernel_3623445/1868502220.py in 13 attacker = Attacker(recipe, dataset, attack_args) 14 ---> 15 attack_results = attacker.attack_dataset() 16

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/attacker.py in attack_dataset(self) 439 self._attack_parallel() 440 else: --> 441 self._attack() 442 443 if self.attack_args.silent:

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/attacker.py in _attack(self) 168 result = self.attack.attack(example, ground_truth_output) 169 except Exception as e: --> 170 raise e 171 if ( 172 isinstance(result, SkippedAttackResult) and self.attack_args.attack_n

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/attacker.py in _attack(self) 166 example.attack_attrs["label_names"] = self.dataset.label_names 167 try: --> 168 result = self.attack.attack(example, ground_truth_output) 169 except Exception as e: 170 raise e

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/attack.py in attack(self, example, ground_truth_output) 421 return SkippedAttackResult(goal_function_result) 422 else: --> 423 result = self._attack(goal_function_result) 424 return result 425

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/attack.py in _attack(self, initial_result) 369 or MaximizedAttackResult. 370 """ --> 371 final_result = self.search_method(initial_result) 372 self.clear_cache() 373 if final_result.goal_status == GoalFunctionResultStatus.SUCCEEDED:

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/search_methods/search_method.py in call(self, initial_result) 34 ) 35 ---> 36 result = self.perform_search(initial_result) 37 # ensure that the number of queries for this GoalFunctionResult is up-to-date 38 result.num_queries = self.goal_function.num_queries

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/search_methods/greedy_word_swap_wir.py in perform_search(self, initial_result) 123 124 # Sort words by order of importance --> 125 index_order, search_over = self._get_index_order(attacked_text) 126 127 i = 0

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/search_methods/greedy_word_swap_wir.py in _get_index_order(self, initial_text) 55 for i in range(len_text) 56 ] ---> 57 leave_one_results, search_over = self.get_goal_results(leave_one_texts) 58 saliency_scores = np.array([result.score for result in leave_one_results]) 59

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/goal_functions/goal_function.py in get_results(self, attacked_text_list, check_skip) 94 attacked_text_list = attacked_text_list[:queries_left] 95 self.num_queries += len(attacked_text_list) ---> 96 model_outputs = self._call_model(attacked_text_list) 97 for attacked_text, raw_output in zip(attacked_text_list, model_outputs): 98 displayed_output = self._get_displayed_output(raw_output)

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/goal_functions/goal_function.py in _call_model(self, attacked_text_list) 214 if text not in self._call_model_cache 215 ] --> 216 outputs = self._call_model_uncached(uncached_list) 217 for text, output in zip(uncached_list, outputs): 218 self._call_model_cache[text] = output

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/goal_functions/goal_function.py in _call_model_uncached(self, attacked_text_list) 163 while i < len(inputs): 164 batch = inputs[i : i + self.batch_size] --> 165 batch_preds = self.model(batch) 166 167 # Some seq-to-seq models will return a single string as a prediction

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/models/wrappers/huggingface_model_wrapper.py in call(self, text_input_list) 53 54 with torch.no_grad(): ---> 55 outputs = self.model(**inputs_dict) 56 57 if isinstance(outputs[0], str):

~/anaconda3/envs/textattack/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, *kwargs) 1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks 1109 or _global_forward_hooks or _global_forward_pre_hooks): -> 1110 return forward_call(input, **kwargs) 1111 # Do not call functions when jit is used 1112 full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/envs/textattack/lib/python3.7/site-packages/transformers/models/bert/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict) 1561 output_attentions=output_attentions, 1562 output_hidden_states=output_hidden_states, -> 1563 return_dict=return_dict, 1564 ) 1565

~/anaconda3/envs/textattack/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, *kwargs) 1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks 1109 or _global_forward_hooks or _global_forward_pre_hooks): -> 1110 return forward_call(input, **kwargs) 1111 # Do not call functions when jit is used 1112 full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/envs/textattack/lib/python3.7/site-packages/transformers/models/bert/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict) 1028 ) 1029 sequence_output = encoder_outputs[0] -> 1030 pooled_output = self.pooler(sequence_output) if self.pooler is not None else None 1031 1032 if not return_dict:

~/anaconda3/envs/textattack/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, *kwargs) 1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks 1109 or _global_forward_hooks or _global_forward_pre_hooks): -> 1110 return forward_call(input, **kwargs) 1111 # Do not call functions when jit is used 1112 full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/envs/textattack/lib/python3.7/site-packages/transformers/models/bert/modeling_bert.py in forward(self, hidden_states) 656 # to the first token. 657 first_token_tensor = hidden_states[:, 0] --> 658 pooled_output = self.dense(first_token_tensor) 659 pooled_output = self.activation(pooled_output) 660 return pooled_output

~/anaconda3/envs/textattack/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, *kwargs) 1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks 1109 or _global_forward_hooks or _global_forward_pre_hooks): -> 1110 return forward_call(input, **kwargs) 1111 # Do not call functions when jit is used 1112 full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/envs/textattack/lib/python3.7/site-packages/torch/nn/modules/linear.py in forward(self, input) 101 102 def forward(self, input: Tensor) -> Tensor: --> 103 return F.linear(input, self.weight, self.bias) 104 105 def extra_repr(self) -> str:

RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc) However, sometimes the code does not report errors and successfully generates Chinese counter samples.So it really bothered me.

[jxmorris12]

Thanks for reporting @buthi! I am still not convinced this is an error in TextAttack– especially because it is non-deterministic. Maybe you can follow the instructions here: https://stackoverflow.com/questions/66600362/runtimeerror-cuda-error-cublas-status-execution-failed-when-calling-cublassge

They recommend to set the environment variable CUDA_LAUNCH_BLOCKING=1 for a more readable error message.

[buthi]

感谢您的报告@buthi！我仍然不相信这是 TextAttack 中的错误——尤其是因为它是不确定的。也许您可以按照此处的说明进行操作：https ://stackoverflow.com/questions/66600362/runtimeerror-cuda-error-cublas-status-execution-failed-when-calling-cublassge

他们建议设置环境变量 CUDA_LAUNCH_BLOCKING=1 以获得更易读的错误消息。

Thank you very much for your answer. I changed the package to CU113 to solve the above problems. But now I have another problem that also puzzles me. the code is: `from tqdm import tqdm from textattack.loggers import CSVLogger from textattack.attack_results import SuccessfulAttackResult from textattack import Attacker from textattack import AttackArgs from textattack.datasets import Dataset

attack_args = AttackArgs(num_examples=-1, log_to_txt='/home/ysc/TextAttack/examples/results/attack')

attacker = Attacker(attack, dataset, attack_args)

attack_results = attacker.attack_dataset() There are 500 texts in the attack. Interrupt and error during program running: --------------------------------------------------------------------------- LangDetectException Traceback (most recent call last) /tmp/ipykernel_3654022/847965657.py in 10 attacker = Attacker(attack, dataset, attack_args) 11 ---> 12 attack_results = attacker.attack_dataset()

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/attacker.py in attack_dataset(self) 439 self._attack_parallel() 440 else: --> 441 self._attack() 442 443 if self.attack_args.silent:

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/attacker.py in _attack(self) 185 pbar.update(1) 186 --> 187 self.attack_log_manager.log_result(result) 188 if not self.attack_args.disable_stdout and not self.attack_args.silent: 189 print("\n")

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/loggers/attack_log_manager.py in log_result(self, result) 44 self.results.append(result) 45 for logger in self.loggers: ---> 46 logger.log_attack_result(result) 47 48 def log_results(self, results):

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/loggers/file_logger.py in log_attack_result(self, result) 52 "-" 45 + " Result " + str(self.num_results) + " " + "-" 45 + "\n" 53 ) ---> 54 self.fout.write(result.str(color_method=self.color_method)) 55 self.fout.write("\n") 56

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/attack_results/attack_result.py in str(self, color_method) 71 72 def str(self, color_method=None): ---> 73 return "\n\n".join(self.str_lines(color_method=color_method)) 74 75 def goal_function_result_str(self, color_method=None):

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/attack_results/attack_result.py in str_lines(self, color_method) 67 representation.""" 68 lines = [self.goal_function_result_str(color_method=color_method)] ---> 69 lines.extend(self.diff_color(color_method)) 70 return lines 71

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/attack_results/attack_result.py in diff_color(self, color_method) 89 t2 = self.perturbed_result.attacked_text 90 ---> 91 if detect(t1.text) == "zh-cn" or detect(t1.text) == "ko": 92 return t1.printable_text(), t2.printable_text() 93

~/anaconda3/envs/textattack/lib/python3.7/site-packages/langdetect/detector_factory.py in detect(text) 128 detector = _factory.create() 129 detector.append(text) --> 130 return detector.detect() 131 132

~/anaconda3/envs/textattack/lib/python3.7/site-packages/langdetect/detector.py in detect(self) 134 which has the highest probability. 135 ''' --> 136 probabilities = self.get_probabilities() 137 if probabilities: 138 return probabilities[0].lang

~/anaconda3/envs/textattack/lib/python3.7/site-packages/langdetect/detector.py in get_probabilities(self) 141 def get_probabilities(self): 142 if self.langprob is None: --> 143 self._detect_block() 144 return self._sort_probability(self.langprob) 145

~/anaconda3/envs/textattack/lib/python3.7/site-packages/langdetect/detector.py in _detect_block(self) 148 ngrams = self._extract_ngrams() 149 if not ngrams: --> 150 raise LangDetectException(ErrorCode.CantDetectError, 'No features in text.') 151 152 self.langprob = [0.0] * len(self.langlist)

LangDetectException: No features in text.

[Succeeded / Failed / Skipped / Total] 99 / 78 / 91 / 268: 54%|███▏ | 269/499 [00:34<00:29, 7.86it/s]`

Or an error: `--------------------------------------------- Result 148 ---------------------------------------------

IndexError Traceback (most recent call last) /tmp/ipykernel_3654022/847965657.py in 10 attacker = Attacker(attack, dataset, attack_args) 11 ---> 12 attack_results = attacker.attack_dataset()

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/attacker.py in attack_dataset(self) 439 self._attack_parallel() 440 else: --> 441 self._attack() 442 443 if self.attack_args.silent:

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/attacker.py in _attack(self) 185 pbar.update(1) 186 --> 187 self.attack_log_manager.log_result(result) 188 if not self.attack_args.disable_stdout and not self.attack_args.silent: 189 print("\n")

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/loggers/attack_log_manager.py in log_result(self, result) 44 self.results.append(result) 45 for logger in self.loggers: ---> 46 logger.log_attack_result(result) 47 48 def log_results(self, results):

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/loggers/file_logger.py in log_attack_result(self, result) 52 "-" 45 + " Result " + str(self.num_results) + " " + "-" 45 + "\n" 53 ) ---> 54 self.fout.write(result.str(color_method=self.color_method)) 55 self.fout.write("\n") 56

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/attack_results/attack_result.py in str(self, color_method) 71 72 def str(self, color_method=None): ---> 73 return "\n\n".join(self.str_lines(color_method=color_method)) 74 75 def goal_function_result_str(self, color_method=None):

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/attack_results/attack_result.py in str_lines(self, color_method) 67 representation.""" 68 lines = [self.goal_function_result_str(color_method=color_method)] ---> 69 lines.extend(self.diff_color(color_method)) 70 return lines 71

~/anaconda3/envs/textattack/lib/python3.7/site-packages/textattack/attack_results/attack_result.py in diff_color(self, color_method) 108 else: 109 w1 = t1.words[t1_idx] --> 110 w2 = t2.words[t2_idx] 111 if w1 == w2: 112 t2_equal_idxs.add(t2_idx)

IndexError: list index out of range

[Succeeded / Failed / Skipped / Total] 50 / 40 / 57 / 147: 30%|█▊ | 148/499 [00:20<00:49, 7.13it/s]`

Repeat running the above code, the program will stop after different results and report the same error.

[jxmorris12]

Thanks! These are real bugs. Sorry, this must be frustrating.

What dataset are you using?

[buthi]

Thank you for your reply. I used a Chinese text dataset I had crawled myself.

[jxmorris12]

@buthi I think your problem is related to the following inconsistency:

from textattack.shared.utils import words_from_text

(Pdb) p words_from_text('professor shih 施春风 fong')
['professor', 'shih', '施', '春风', 'fong']
(Pdb) p words_from_text("professor shih choon fong ( 施春风 ; born 1945 ) is the former president of the king abdullah university")
['professor', 'shih', 'choon', 'fong', '施春风', 'born', '1945', 'is', 'the', 'former', 'president', 'of', 'the', 'king', 'abdullah', 'university']
(Pdb) p words_from_text(' 施春风')
['施', '春风']

Our word-replacement logic depends on the fact that calling words_from_text on a substring will produce the same words as calling it on the full string. That's not the case, so it malfunctions here and breaks our indexing map.

Let me know if this helps you begin to debug.

[liuyuyan2717]

首先确认你的python库版本正确，如果仍然出现问题，应该是显存。 解决方法一：然后你用自己的模型在中文数据集上32G的显存是不能跑32batch的，去修改Textattack\textattack\models\wrappers\pytorch_model_wrapper.py下的batchsize。 解决方法二：一般你的句子不会太长，textattack默认用了512长度，你可以考虑截断一下，这样会非常省显存：